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DEPLOYED  ANALYST  HANDBOOK 
(DAHB) 

JUNE  2016 

SUMMARY 

THE  PROJECT  PURPOSE:  This  handbook  is  intended  to  be  a  quiek  referenee  guide  for  all 
deploying  Operations  Researeh/Systems  Analysis  (ORSA)  analysts,  whether  military,  eivilian,  or 
eontraetor. 

THE  PROJECT  SPONSOR:  Functional  Area  (FA)  49  Proponency  Office. 

THE  PROJECT  OBJECTIVES: 

Present  ORSA  analysts  with  a  wide  range  of  topics  from  specific  analytical  products  to  ways  to 
prepare  and  communicate  analyses. 

Present  examples  of  analyses  developed  in  theater  and  through  reachback  support. 

Present  examples  of  analyses  that  ORSA  analysts  can  expect  to  execute  in  a  combat  theater. 

Provide  references  for  ORSA  analysts  who  are  interested  in  additional  resources  and  examples. 

THE  SCOPE  OF  THE  PROJECT:  The  information  contained  in  this  handbook  was 
developed  from  the  experiences  of  ORSA  analysts  who  deployed  to  operational  theaters  in  Iraq, 
Afghanistan,  the  Philippines,  the  Horn  of  Africa,  the  Middle  East,  and  elsewhere  over  the  past 
14-plus  years. 

COMMENTS  AND  QUESTIONS  may  be  sent  to  the  Director,  Center  for  Army  Analysis, 
ATTN:  CSCA-OA,  6001  Goethals  Road,  Suite  102,  Fort  Belvoir,  VA  22060-5230 
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Dedication.  This  Deployed  Analyst  Handbook  (DAHB)  is  dedicated  to  the  142  ORSA  analysts 
who  deployed  under  Center  for  Army  Analysis  (CAA)  auspices  to  Iraq,  Afghanistan,  the  Horn  of 
Africa,  and  the  Philippines  from  2002  through  June  of  2016. 

The  Center  would  also  like  to  thank  all  Operations  Research/Systems  Analysis  (ORSA)  analysts 
who  contributed  to  the  development  of  the  DAHB,  as  well  as  reviewing  agencies,  to  include  the 
U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA),  the  Marine  Corps  Combat 
Development  Command  (MCCDC),  the  U.S.  Army  Training  and  Doctrine  Command  Analysis 
Center  (TRAC),  and  the  Functional  Area  (FA)  49  Proponency  Office. 

Intended  Audience.  Military  FA  49s,  civilian  career  program  (CP)  36/15 15s,  and  ORSA 
contractors  who  directly  support  the  warfighter  at  brigade  and  higher  levels  of  command. 

Reference  Guide.  The  DAHB  is  a  quick  reference  guide  for  all  deploying  ORSA  analysts  and 
contains  a  wide  range  of  techniques  and  analytical  products  that  ORSA  analysts  have  developed 
in  theater  and  through  reachback  support,  and  have  used  in  operational  theaters.  The  lessons 
learned  and  best  practices  contained  in  the  DAHB  were  consolidated  from  numerous  ORSA 
analysts  who  have  deployed  in  support  of  Operations  Iraqi  Freedom  (OIF)  and  New  Dawn 
(OND)  in  Iraq;  Operations  Enduring  Freedom  (OEF)  and  Ereedom's  Sentinel  (OFS)  in 
Afghanistan;  Operation  Inherent  Resolve  (OIR)  in  the  Eevant,  and  other  Overseas  Contingency 
Operations  (OCOs)  in  the  Philippines,  the  Horn  of  Africa,  and  elsewhere  around  the  world  over 
the  past  14-plus  years.  The  DAHB  is  informative,  relevant,  and  full  of  information — not  easily 
obtained  elsewhere — that  will  prepare  deploying  ORSA  analysts  for  their  unique  and  highly 
critical  assignments  at  the  tactical,  operational,  and  strategic  levels  of  war. 

Topic  Selections.  The  information  and  examples  presented  in  this  handbook  highlight  the  types 
of  analyses  that  ORSA  analysts  can  expect  to  conduct  in  the  future.  CAA  thoughtfully  chose 
these  specific  examples  through  working  groups  and  advice  received  from  experts  throughout  the 
analytical  community. 

Motivation  for  Using  Handbook.  This  handbook  complements  the  CAA  Deploying  Analyst 
Course  (DAC)  by  providing  a  companion  text  that  assists  deploying  ORSA  analysts  who  are 
junior  military  officers,  civilians,  and  contractors.  This  handbook  helps  prepare  ORSA  analysts 
to  provide  analytical  support  to  their  respective  commands  and  staffs  by  exposing  them  to  best 
practices  and  lessons  learned,  and  informing  them  of  the  reachback  processes. 
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Additional  Deployed  Analyst  Information  and  Insights: 

•  CAA  Deployed  Analyst  History  Report  Vol.  I  (Analytic  Support  to  Combat  Operations  in 
Iraq  (2002  -  201 1)),  March  2012,  available  at 

http://www.caa.army.mil/DAHR%20Volume%201%20-%20Iraq%20Final- 
March%2020 1 2.pdf. 

•  CAA  Deployed  Analyst  History  Report  (DAHR)  Vol.  II  (Analytic  Support  to  Combat 
Operations  in  The  Philippines  (2011  -  2014)),  December  2015,  available  at 
http://www.caa.army.mil/DAHR-Philippines(Final%20Report)2015 1203.pdf. 
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CHAPTER  1  COMMUNICATING  ORSA  CAPABILITIES 

1.1  What  is  Operations  Research? 


General,  What  is  Operations  Research/Systems  Analysis?  It  is  difficult  to  answer  this  question 
concisely  without  using  words  like  stochastic,  combinatorial,  and  heteroscedasticity.  However, 
effectively  communicating  one's  unique  ORSA  capabilities  to  the  command  is  as  critical  to  one's 
success  as  is  the  quality  of  one's  work. 

In  a  deployed  environment,  ORSA  analysts  often  do  not  have  the  luxury  of  external  resources  to 
assist  in  communicating  their  skill  set  to  the  command  and  others  who  would  benefit  from  their 
assistance.  Many  or  most  ORSA  analysts  are  introverted  by  nature.  More  often  than  not, 
socializing  one's  skill  set  does  not  come  naturally.  However,  it  is  a  talent  that  can  be  developed 
with  practice. 

This  chapter  offers  techniques  for  ORSA  analysts  to  use  (1)  in  explaining  their  profession 
concisely  to  others,  (2)  in  applying  their  skills  in  a  deployed  environment,  and  (3)  in  effectively 
presenting  their  work.  Much  of  the  content  of  this  chapter  parallels  information  from  the 
Institute  for  Operations  Research  and  the  Management  Sciences  (INFORMS)  on  communicating 
the  ORSA  profession. 

1.2  Concisely  Communicating  the  ORSA  Profession  to  Others 


ORSA  Defined.  ORSA  is  the  acronym  for  the  scientific  field  of  "Operations  Research/Systems 
Analysis,"  but  ORSA  is  often  used  colloquially  to  mean  an  "ORSA  analyst."  Introducing  oneself 
as  an  ORSA  peaks  the  attention  of  others  and  leads  to  further  inquiry.  As  senior  leaders  become 
familiar  with  the  highly  skilled,  academic  nature  of  an  ORSA  analyst's  work,  they  gain  a  sober 
appreciation  for  the  tremendous  value  ORSA  analysts  bring  to  the  fight.  In  short  order,  senior 
leaders  associate  ORSA  analysts  with  insight  and  solutions  for  solving  complex  problems  at  the 
tactical,  operational,  and  strategic  levels  of  war. 

Responsibilities  of  a  deployed  ORSAfi 

ORSAs  use  analytical  methods  and  mathematically  based  procedures  to  enable  leadership  decisions 
in  a  constantly  changing  global  environment.  [ORSAs]  introduce  quantitative  and  qualitative 
analysis  to  the  military's  decision-making  processes  by  developing  and  applying  probability  models, 
statistical  inference,  simulations,  optimization,  and  economic  models.  [ORSAs  work  in]  diverse 
disciplines  that  include  personnel  management,  doctrine  and  force  development,  training 
management,  system  testing,  system  acquisition,  decision  analysis,  and  resource  management,  as 
well  as  tactical,  operational  and  strategic  planning  from  division  through  combatant  command,  and 
from  Army  Command  (ACOM)  through  the  highest  levels  of  the  [Department  of  Defense]  DOD 
(Department  of  the  Army  Pamphlet  [DA  PAM]  600-3,  2014,  p.  310). 

ORSAs  integrate  military  knowledge  and  experienee  with  the  seientific  and  managerial  fields. 
They  serve  as  subject  matter  experts  in  designing  forces,  allocating  resources,  analyzing  effects. 


1  This  discussion  about  the  responsibilities  of  a  deployed  Army  Officer  ORSA  taken  from  DA 
PAM  600-3,  2014  is  substantially  applicable  to  deployed  Army,  sister  service,  and  partner  nations’ 
military,  civilian,  and  contractor  ORSAs. 
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performing  course  of  action  and  trade-off  analysis,  and  they  effectively  communicate  potential 
solutions  to  complex  problems  to  decision  makers.  The  ORSA  officer  will  typically  serve  in  one 
of  several  general  assignments: 

(1)  As  a  combat  analyst  on  a  division,  corps.  Army  Service  Component  Command  (ASCC),  (or 
equivalent  joint  headquarters  staff),  or  Combatant  Command  headquarters  staff 

(2)  As  an  analyst  on  an  Army,  joint,  or  defense  agency  staff,  a  field-operating  agency  (FOA)  or  a 
direct  reporting  unit,  (e.g.,  CAA,  Human  Resources  Command  (HRC),  or  the  Army  Test  and 
Evaluation  Command). 

(3)  As  an  analyst  in  an  ACOM,  i.e.,  U.S.  Army  Training  and  Doctrine  Command  (TRADOC), 
U.S.  Army  Forces  Command  (FORSCOM),  or  U.S.  Army  Materiel  Command  (AMC),  either  in 
the  command  headquarters  or  in  an  organization  whose  principal  mission  is  to  provide  analysis 
that  supports  the  organizing,  equipping,  manning,  training  and  operations  of  military  forces. 

Such  organizations  include  the  TRADOC  Army  Capabilities  Integration  Center  (ARCIC), 
TRADOC  Analysis  Center  (TRAC),  and  Army  Centers  of  Excellence  (CoE). 

(4)  As  an  instructor  teaching  Operations  Research/Systems  Analysis  and/or  mathematics  courses 
at  the  United  States  Military  Academy  (USMA),  Naval  Postgraduate  School  (NPS),  Air  Force 
Institute  of  Technology  or  a  TRADOC  School.  (DA  PAM)  600-3,  2014,  p.  311). 

The  ORSA  Skill  Set,  ORSA  analysts  (1)  investigate  essential  factors  that  illuminate 
understanding;  (2)  communicate  complex  ideas  to  decision  makers  at  brigade  and  higher  levels 
of  command;  and  (3)  recommend  solutions  to  complex  problems. 

ORSA  analysts  apply  their  skill  set  through  (1)  problem  analysis,  using  various  analytical 
techniques;  (2)  probability  and  statistics  analysis  to  gain  valuable  insight;  and  (3)  optimization 
analysis  to  quantify  options  and  select  the  best  option  for  a  complex  problem  that  has  multiple 
solution  sets. 

Application  examples  include 

•  analyzing  significant  event  data  in  order  to  identify  spatial  and  temporal  trends; 

•  developing  strategies  to  effectively  allocate  or  employ  limited  resources; 

•  improving  supply  chain  and  logistics  operations; 

•  conducting  qualitative  and  quantitative  assessments  of  current  operations; 

•  measuring  risk  and  uncovering  factors  critical  to  managing  and  reducing  risk;  and 

•  evaluating  the  potential  benefit  of  changes  to  tactics,  techniques,  and  procedures  (TTPs). 

1.3  How  to  Recognize  When  Others  Would  Benefit  from  ORSA 


An  ORSA  analyst  can  determine  when  others  would  benefit  from  their  skill  set  through  a  variety 
of  indicators  including  identifying  staff 

•  with  a  large  amount  of  data  but  do  not  know  what  is  important  or  how  to  use  it; 

•  who  are  responsible  for  inputting  and  storing  their  data  in  Excel  but  do  not  have  the  skills 
to  do  so; 
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•  who  do  not  know  how  to  display  their  data  to  senior  leaders; 

•  who  are  dealing  with  a  complex  problem  with  many  decision  points  and  need  a  method  to 
analytically  evaluate  their  options  and  choose  the  best  one; 

•  who  are  risk-averse  and  do  not  have  effective  strategies  for  limiting,  reducing,  or 
managing  risk;  or 

•  who  have  variables  they  want  to  measure  but  do  not  know  how  to  measure  them. 

1.4  Valuable  Applications  of  the  ORSA  Skill  Set  in  a  Deployed  Environment 


Most  ORSA  analysts  do  not  have  the  time  required  to  use  advanced  ORSA  techniques  in  a 
deployed  environment;  however,  there  are  less  complex  applications  that  are  highly  valuable  to 
theater  commands.  These  include 

•  managing,  analyzing,  and  displaying  data; 

•  organizing  large  amounts  of  data; 

•  communicating  technical  information  to  commanders  and  senior  leaders; 

•  thinking  critically,  i.e.,  "outside  the  box",  and  developing  strategies  to  solve  complex 
problems; 

•  advocating  change  and  providing  clear  answers  to  problems  others  are  reluctant  to  tackle; 

•  developing  metrics  to  measure  campaign  progress; 

•  providing  recommendations  that  are  tactically,  operationally,  and  strategically  relevant; 
and 

•  teaching  basic  and  advanced  Excel  and  PowerPoint  skills. 

In  the  ORSA  community,  these  tools  are  often  taken  for  granted,  e.g.,  pivot  tables  and  charts; 
linking  PowerPoint  data  to  external  statistical  databases;  or  using  Visual  Basic  for  Applications 
(VBA)  to  automate  tasks.  The  ORSA  skill  set  is  in  high  demand  and  pays  huge  dividends  for 
theater  commanders.  For  example,  teaching  staff  members  Excel  technical  shortcuts  saves  them 
a  great  deal  of  time  (the  most  highly  valued  asset  in  theater).  There  is  a  saying,  "If  you  give  a 
man  a  fish,  you  feed  him  for  a  day.  If  you  teach  a  man  to  fish,  you  feed  him  for  a  lifetime" 
(author  unknown). 

1.5  Ensuring  One's  Relevancy 


ORSA  analysts  often  must  establish  their  role  with  other  staff  officers  in  the  unit  who  may  be 
confused  as  to  the  nature  of  their  duties.  This  is  essential  in  ensuring  that  ORSA  analysts  are 
used  appropriately  for  the  services  they  offer.  Listed  below  are  techniques  incoming  ORSA 
analysts  should  consider  when  establishing  their  role. 

Keep  others  informed.  ORSA  analysts  should  strive  to  become  extroverted  collaborators,  not 
"isolationists."  They  should  be  excited  about  their  work  and  share  it  with  others.  Products 
improve  when  one  collaborates  with  others  and  involves  them  in  the  process.  No  matter  the 
level  of  expertise,  others  have  opinions,  unique  histories  and  backgrounds,  and  "an  extra  set  of 
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eyes"  to  enhanee  the  process.  Others  cannot  help  solve  a  problem  or  assist  with  a  project  unless 
they  are  aware  of  it. 

Make  time  for  face-to-face  interaction  with  others.  In  today's  digital  world,  staff  work  can 
keep  professionals  at  their  desks  for  hours  on  end.  ORSA  analysts  should  get  out  and  mingle 
with  their  counterparts.  This  is  the  best  way  for  everyone  to  get  to  know  each  other  and  share 
problems,  issues,  concerns,  and  ideas.  Before  one  can  influence  others,  one  must  gain  their  trust 
and  respect.  ORSA  is  a  quiet  profession.  ORSA  analysts  have  to  get  out  and  make  a  little  noise 
if  they  want  to  be  utilized.  As  a  rule-of-thumb,  one  should  spend  more  time  interacting  with 
others  face-to-face  than  interacting  with  others  over  the  computer. 

Do  not  emphasize  problems.  Commanders  and  staff  appreciate  solutions.  Problems  are  just 
challenges  waiting  to  be  met  by  determined  ORSA  analysts  and  their  in  and  out  of  theater 
networks  of  associates.  This  does  not  mean  the  ORSA  analyst  is  an  expert  in  every  staff  issue. 
However,  the  ORSA  analyst  should  talk  to  staff  members  to  uncover  solutions  that  can  serve  to 
solve  problems. 

Create  the  opportunity  for  follow-on  interaction.  When  an  ORSA  analyst  meets  people  face- 
to-face  to  discuss  issues,  he  or  she  should  set  a  time  to  meet  again.  This  helps  keep  the 
momentum  going  so  others  do  not  lose  interest  or  become  sidetracked.  One  should  say 
something  like,  "when  we  meet  again  at  1000  on  Thursday,  we  will  discuss  progress  on  x,  y,  and 
z." 

Actively  seek  out  resources,  e.g.,  colleagues,  mentors,  and  "believers."  There  are  people  in  the 
command  who  have  ORSA  backgrounds  and/or  are  aware  of  the  value  ORSA  analysts  bring  to 
the  table.  One  should  develop  a  strong  rapport  with  these  people,  keep  them  informed  of  current 
work,  and  ask  these  people  if  they  are  willing  to  collaborate  on  projects.  This  is  especially 
important  if  one  is  a  "lone"  deployed  ORSA  analyst.  ORSA  analysts  should  find  others  who 
appreciate  groundbreaking  ideas  and  support  "out  of  the  box"  projects. 

Familiarize  others  with  ORSA  work.  The  following  technique  can  increase  one's  exposure: 
When  briefing  or  describing  one's  work  to  others,  say,  "Do  you  think  Colonel  Smith  would  like 
to  see  this  analysis?"  When  one  receives  an  affirmative  answer,  one  has  a  perfectly  good  reason 
to  walk  into  Colonel  Smith's  office  and  present  one's  great  work.  When  possible,  an  ORSA 
analyst  should  bring  along  his  or  her  supervisor  or  counterpart  in  or  near  Colonel  Smith's  office 
or  network.  This  builds  relationships  within  an  ORSA  analyst's  circle  of  influence. 

Start  with  projects  most  applicahle  and  important  to  the  command.  ORSA  analysts  could 
spend  months  simply  mining  through  data  sets.  As  a  deployed  ORSA  analyst,  one  does  not  have 
the  luxury  of  time.  Deployed  ORSA  analysts  need  to  produce  results  in  days  (or  sometimes 
hours).  Good  work  completed  quickly  is  better  than  great  work  completed  late.  People  need  to 
receive  answers  while  they  still  remember  their  questions.  Conducting  timely  analysis  that  is 
most  applicable  and  important  to  the  command  creates  a  healthy  respect  for  the  ORSA 
profession.  Once  senior  leaders  understand  and  appreciate  the  ORSA  skill  set,  they  will  seek  out 
ORSA  analyst  assistance  on  complex  problems.  By  solving  problems  quickly  and  well,  one 
becomes  the  go-to  person  in  the  command  for  advice  and  for  solutions. 

Answer  the  question.  ORSA  analysts  should  resist  the  temptation  to  over  promise.  When 
presented  with  a  specific  question,  one  should  always  answer  that  question  first,  especially  prior 
to  explaining  to  the  senior  leader  that  the  question  he  asked  is  not  the  "real"  question  at  hand.  If 
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time  permits,  one  can  conduct  further  analysis  and  provide  alternative  solutions  and 
recommendations.  Be  humble  when  explaining  new  concepts  to  senior  leaders;  for  example, 
introducing  the  concept  of  normalizing  casualty  statistics  against  the  population  of  a  security 
service,  e.g.,  52  killed  in  action/lOk  troops.  Be  prepared  to  explain  what  you  mean  in  clear 
simple  language  and  how  normalizing  casualties  provides  context  not  found  in  the  raw  numbers. 

Prepare  concise  reports  and  clean  presentations.  Two  or  three  pages  or  slides  are  usually 
sufficient.  When  possible,  an  ORSA  analyst  should  present  his  or  her  results  in  one  page  or 
slide.  One  should  write  in  Standard  English  so  everyone  can  understand.  Presentations  should 
be  brief  and  direct.  One  should  highlight  or  bold  the  insights/results  of  the  analysis. 

Do  not  sell  the  algorithm;  sell  the  results.  Hardly  anyone  cares  that  one  performed  a 
Chi-Squared  test  with  41  degrees  of  freedom  or  a  logistic  regression  with  12  independent 
variables  with  interactions.  An  ORSA  analyst  should  present  findings/results  of  the  analysis  and 
their  significance  to  the  mission.  Too  many  ORSA  analysts  spend  too  much  time  writing  and 
talking  about  how  they  did  the  analysis.  Senior  leaders  do  not  need  the  process;  they  need  the 
results.  One  should  share  his  or  her  conclusions  and  make  recommendations. 

1.6  Summary 


This  chapter  explained  the  role  of  the  ORSA  analyst.  It  was  designed  to  teach  a  deploying 
ORSA  analyst  how  to  (1)  communicate  his  or  her  profession  to  others;  (2)  apply  his  or  her  skill 
set  in  a  deployed  environment;  and  (3)  become  the  go-to  person  in  his  or  her  command.  Good 
communication  skills  are  vital  to  one's  success  as  a  deployed  ORSA  analyst.  Effective 
communication  does  not  replace  sound  analytical  work;  both  are  required  in  a  deployed 
environment. 
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CHAPTER  2  DATA  MANAGEMENT  METHODS 

2.1  Why  Data  Management? 


Data  Management  is  the  proeess  of  planning,  coordinating,  and  controlling  an  organization's  data 
resources.  Data  management  is  generally  a  Chief  Information  Office  (CIO)/  G-6  function; 
however,  in  a  deployed  environment,  ORSA  analysts  play  a  significant  role  in  data  management. 
Key  responsibilities  associated  with  data  management  include  data  acquisition/collection,  data 
indexing/cataloging/storage,  and  data  verification/validation. 

This  chapter  provides  an  overview  of  data  management  methods.  It  is  not  a  cookbook  of 
detailed  procedures  or  an  exploration  of  abstract,  conceptual  techniques.  Rather,  it  provides  an 
analytical  approach  that  is  responsive  to  the  needs  of  decision  makers  at  the  highest  levels  of 
military  command.  It  includes  data  management  topics  ranging  from  general  data  considerations 
to  specific  data  requirements:  data  collection,  data  entry,  data  reporting,  and  data  analysis. 

2.2  Data  Management  Goals  and  Objectives 


ORSA  analysts  seldom  render  advice  based  on  perfect  information.  An  ORSA  analyst's  task  is 
to  discern  which  data  are  pertinent  and  meaningful  and  then  use  these  data  in  the  problem 
solving  process.  Inappropriate,  inaccurate,  and  invalid  data  are  sources  of  fruitless  analytical 
effort  (sound  study  efforts  based  on  sound  analytical  designs  are  of  little  value  without  valid 
data).  It  is  a  rookie  mistake  to  base  judgments  on  the  unchallenged  use  of  official  figures.  One 
must  challenge  his  or  her  sources  of  information.  One  must  examine  the  data  that  researchers 
used  to  reach  their  conclusions.  One  must  determine  how  researchers  derived  their  findings,  and 
one  must  check  data  for  accuracy.  A  key  function  of  analysis  is  to  reject  bad  data  and  inaccurate 
information.  Ensuring  and  maintaining  data  integrity  is  fundamental  to  the  military  mission  and 
requires  a  considerable  investment  of  time  and  energy. 

2.3  Data  Management  Process  Overview 


Data  Flow  Model.  A  simple  data  flow  model  provides  a  good  starting  point  for  understanding 
how  to  carry  out  data  handling  procedures.  Figure  1  identifies  five  key  steps  in  data  flow: 
acquisition,  verification,  validation,  analysis,  and  dissemination.  These  steps — plus  additional 
procedural  details — are  described  in  the  following  pages.  Additionally,  storage,  maintenance, 
and  security  issues  apply  at  all  stages  of  data  flow. 

Storage,  Maintenance,  Security 

- ► 


Acquisition 

Verification 

Validation 

Analysis 

Dissemination 

Figure  1.  Steps  in  data  flow  model. 

Data  Acquisition.  Acquiring  and  collecting  data  from  many  internal  and  external  sources  is 
critical  and  often  complicated.  Sometimes  there  is  a  need  to  collect  raw  data  in  the  field,  such  as 
post  blast  details  from  an  improvised  explosive  device  (lED)  event.  Other  times,  there  is  a  need 
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to  elicit  data  from  other  people,  e.g.,  polling/survey  data.  Data  may  be  acquired  through  data 
mining  or  field  studies  conducted  personally  or  by  other  analysts.  Regardless  of  one's  method  of 
collection,  one  must  filter  and  validate  the  data  to  avoid  garbage  in,  garbage  out.  Data  quality  is 
an  extremely  important  issue. 

2.4  Three  Common  Characteristics  of  Data  Collection 


Data  Population  for  Measurement  (Who?)  In  order  to  choose  the  most  appropriate  statistical 
methods  of  analysis,  one  must  precisely  define  the  population  he  or  she  wishes  to  measure.  The 
specific  segments  of  the  general  population  one  samples  are  called  sampling  units.  Sampling 
units  must  be  unique,  easily  identifiable,  and  selectable.  For  instance,  if  a  poll  involves  Soldiers 
from  multiple  units,  the  researcher  must  capture  certain  criteria  associated  with  each  Soldier, 
e.g.,  rank,  branch,  duties,  and  responsibilities,  so  he  or  she  can  easily  identify  all  measurements 
from  this  population.  If  the  measured  population  includes  Private  Jones,  an  Infantry  Soldier  with 
the  101st  Airborne  Division  who  served  in  Afghanistan  from  May  2014  to  May  2015,  the 
researcher  must  identify  measurements  associated  with  this  Soldier,  i.e.,  personal  characteristics 
and  professional  activities.  Sampling  units  must  be  both  discrete  (countable)  and  measurable 
using  standardized  methodologies. 

Items  for  Measurement  (What?)  After  one  defines  the  population  and  sampling  units,  one 
must  define  and  select  the  specific  data  he  or  she  wishes  to  measure.  Measured  data  must  have 
clear  definitions,  recognizable  characteristics,  a  clearly  defined  set  of  observable  measurement 
parameters  (qualification  standards),  and  a  discrete  name.  For  example,  when  a  soldier  submits 
an  initial  spot  report,  he  or  she  describes  enemy  activity,  gives  a  clear  description  of  what  the 
friendly  forces  were  doing  at  the  time  of  the  incident  (patrol,  recon,  etc.),  pinpoints  where  the 
incident  took  place  (grid  location),  how  many  soldiers  were  involved  in  the  incident,  what  kind 
of  incident  occurred,  e.g.,  lED  explosion,  and  what  the  outcome  of  the  incident  was,  i.e.,  battle 
damage  assessment.  One  should  treat  each  of  these  data  points  as  discrete  pieces  of  the  event  to 
facilitate  further  analysis  and  comparison  with  future  or  past  events  that  are  similar. 

Methods  of  Performing  the  Measurement  (How?)  Data  are  generally  classified  as  objective 
or  subjective,  depending  on  the  amount  of  human  judgment  involved  with  measuring  them. 
Objective  data  have  physical/tangible  properties,  e.g.,  grid  location,  speed  of  a  vehicle,  number 
of  vehicles  in  a  convoy  and  require  little  or  no  human  judgment.  On  the  other  hand,  subjective 
data,  e.g.,  lessons  learned  captured  from  surveys  submitted  by  redeploying  soldiers,  require 
human  judgment.  For  some  aspects  of  data  collection,  objective  data  may  ultimately  give 
sensitive,  reliable,  and  valid  results.  Other  data,  such  as  soldier  feedback  and  assessments, 
require  subjective  measurements  and  interpretation.  A  researcher  must  evaluate,  score,  and 
assign  a  rating  to  each  subjective  measurement.  He  or  she  must  assess  and  measure  the  validity 
of  each.  In  every  case — to  the  greatest  extent  possible — a  researcher  must  minimize  his  or  her 
personal  biases  in  order  to  ensure  valid  and  reliable  data  collection. 

2.5  High  Quality  Data  Attributes 


Sensitivity.  Sensitivity  means  the  extent  to  which  a  small  change  in  a  parameter  results  in  a 
relatively  large  change  in  the  outcome.  If  a  small  change  in  a  parameter  results  in  a  relatively 
large  change  in  the  outcome,  the  outcome  is  said  to  be  sensitive  to  that  parameter. 
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Reliability.  Reliability  means  free  from  random  error  in  the  final  measurement,  usually 
indieated  by  eonsisteney  among  items  or  stability  in  the  measurements  over  time.  Both  sampling 
and  non-sampling  errors  affeet  reliability.  Sampling  errors  are  the  inevitable  result  of  basing  an 
inferenee  on  a  random  sample  rather  than  on  the  entire  population.  Non-sampling  errors  oeeur 
from  non-response,  ending  errors,  eomputer  proeessing  errors,  errors  in  the  sampling  frame, 
reporting  errors,  etc. 

Validity.  Validity  means  that  the  data  accurately  represent  what  they  are  intended  to  represent. 
Data  validity  depends  partly  on  having  adequate  sensitivity  and  reliability  of  the  measures; 
however,  even  with  adequate  sensitivity  and  reliability,  the  data  may  represent  something  other 
than  what  they  appear  to  represent.  For  example,  a  poorly  worded  survey  question  may  yield 
results  that  do  not  accurately  answer  the  desired  question  (invalid  results).  Vague  data 
definitions,  insufficient  interviewer  training,  casual  data  collection  methods,  and  operator 
discretion  weaken  or  reduce  data  validity. 

Standardization.  Standardization  means  ensuring  uniformity  in  data  collection  procedures. 
Standardization  ensures  that  data  differences  are  not  procedural  differences.  The  standardization 
attribute  is  very  sensitive  to  non-sampling  errors  associated  with  quality  of  data,  collection 
techniques,  and  the  stability  of  established  data  collection  procedures. 

Completeness.  Completeness  means  that  all  the  expected  data  elements,  records,  observations, 
etc.  are  present.  Non-sampling  errors  affect  completeness.  Common  causes  of  incomplete  data 
include  (a)  failure  to  enforce  data  collection  standards  outlined  in  the  standardized  data  entry 
report,  (b)  failure  to  report  all  of  the  required  information,  (c)  failure  to  secure  records  from 
which  the  data  are  to  be  collected,  (e)  inability  to  read/scan  the  data  collection  form,  and  (f) 
corrupted  fdes. 

2.6  Data  Entry 


Data  entry  is  the  initial  set  of  operations  used  to  transcribe  the  raw  source  data  (survey  results, 
initial  spot  reports,  etc.)  into  a  computerized  format,  i.e.,  a  database.  Data  entry  begins  when 
data  collection  is  complete.  Data  entry  forms  and  quality  control  features  are  extremely 
important  in  minimizing  error.  One  should  standardize  data  entry  forms  across  all  levels  of 
command  to  ensure  consistency  in  data  collection.  Standardized  forms  should  contain  logical 
formats  that  include  discrete  data  entry  choices  e.g.,  drop  down  menus,  to  reduce  transcription 
errors.  Inevitably,  the  process  of  transcribing  data  from  field  forms  to  a  digital  format  introduces 
error.  To  minimize  errors,  one  must  standardize  data  entry  forms  and  quality  control  measures 
throughout  the  data  management  process.  For  example,  data  entry  forms  reduce  transcription 
errors  through  pick  lists,  value  limits,  and  controlled  access  to  the  database.  Forms  set  for  "data 
entry  only"  prevent  accidental  deletion  or  alteration  of  existing  data.  One  can  also  set  key  fields 
to  prevent  duplicate  entries  such  as  variations  of  a  unit's  name.  With  standardized  forms,  one 
can  also  control  the  sequence  of  data  entry.  A  well-developed  data  entry  format  enables  users  to 
search  for  and  retrieve  consistent  results  that  are  free  from  duplicate  entries. 

2.7  Data  Verification 


Irrespective  of  how  one  acquires  the  data,  one  must  import  the  data  into  the  database  using  a 
process  that  ensures  verification.  Data  verification  immediately  follows  data  entry  and  involves 
checking  the  accuracy  of  source  data.  While  one  rarely  achieves  100%  error-free  data  entry,  one 
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can  minimize  transcription  errors  by  using  a  layered  approach  to  data  verification.  One  should 
use  quality  control  measures  throughout  the  data  management  process,  from  the  entry  of  an 
initial  report,  to  the  consolidation  of  massed  reports  into  an  official  database,  to  retrieval  at 
higher  headquarters  where  one  uses  the  data  for  analysis  and  decision  making.  A  good  rule  of 
thumb  is  for  one  to  verify  10%  of  all  records  for  a  specified  period,  focusing  on  key  aspects  of 
the  report  e.g.,  event  location,  event  category,  etc.  One  should  compare  these  audited  records 
with  the  existing  record  and  include  this  information  in  the  finalized  report.  If  one  finds 
reporting  errors,  he  or  she  should  review  and  re -verify  the  entire  data  set.  Once  one  verifies  that 
the  computerized  data  accurately  reflect  the  original  field  data,  he  or  she  can  index  and  archive 
the  reports  for  later  retrieval. 

2.8  Data  Validation 


Data  validation  ensures  the  accuracy  of  the  original  source  field  reports.  Data  should  be  accurate 
and  logical.  For  example,  a  report  stating  the  convoy  speed  was  150  mph  is  illogical  and  almost 
certainly  incorrect,  whether  or  not  the  original  recorder  properly  transcribed  the  data  from  field 
reports.  As  another  example  of  validated  data,  spatial  data  collected  within  a  certain  boundary 
should  appear  within  that  boundary  when  viewed  in  a  geographic  information  system  (GIS) 
environment.  The  process  of  reviewing  computerized  data  for  range  and  logic  errors  is  the 
validation  stage. 

Database  developers  build  certain  components  of  data  validation  into  their  programs,  e.g.,  range 
limits.  ORSA  analysts  can  perform  additional  data  validation  during  verification  if  they  are 
sufficiently  knowledgeable  about  the  data.  During  data  validation,  an  ORSA  analyst  should 
identify  generic  errors,  e.g.,  missing,  mismatched,  or  duplicate  records,  as  well  as  errors  specific 
to  particular  items  and  types  of  activities.  For  example,  validation  of  enemy  threat  data  includes 
database  queries  and  comparison  of  data  over  different  years.  One  query  may  detect  records  of 
specific  location  identification  (ID)  and  a  certain  type  of  enemy  activity,  e.g.,  lED  attack. 

Another  query  may  count  the  number  of  lED  attack  plots  per  sample  site  to  ensure  that  personnel 
entered  all  plots  correctly.  Additionally,  an  ORSA  analyst  can  compare  current  data  with 
previous  years  to  identify  gross  differences.  One  should  ensure  consistency  between  field  forms 
and  the  database  by  noting  how  and  why  personnel  made  changes  to  the  data  on  the  original  field 
forms.  Once  data  validation  is  complete,  the  data  can  be  archived  for  later  retrieval. 

2.9  Data  Analysis 


As  described  by  Albright,  Winston,  and  Zappe  (2011),  there  are  seven  basic  steps  to  data 
analysis:  (1)  define  the  problem,  (2)  collect  and  summarize  the  data,  (3)  formulate  a  model,  (4) 
verify  the  model,  (5)  select  one  or  more  suitable  decisions,  (6)  present  results,  and  (7)  update  the 
model  as  new  data  become  available.  These  steps  aid  in  planning  the  proper  course  of  data 
analysis.  Simply  put,  there  are  a  myriad  of  statistical  tests,  optimization  methods,  and  simulation 
methods,  and  one  must  determine  which  technique  is  appropriate  for  a  given  situation. 

Analytical  tools  used  to  assist  in  the  process  include,  but  are  not  limited  to.  Excel,  Access,  ESRI 
Arc  Geographical  Information  System  (ArcGlS),  SAS  Institute  Statistical  Analysis  System 
(SAS),  and  IBM's  Statistical  Package  for  the  Social  Sciences  (SPSS).^  All  data  analysis 
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techniques  should  comport  to  scientific  and  professional  standards.  One  should  analyze  results 
for  statistical  significant. 

2.10  Data  Dissemination 


One  should  follow  the  appropriate  review  process  and  share  results  with  all  identified  users.  One 
should  display  data  in  a  logical  and  consistent  format  and  follow  ownership  protocol  guidelines. 
To  accomplish  this  goal,  one  should  use  standardized  procedures  to  quality-check,  analyze, 
document,  catalogue,  archive,  and  make  data  available  for  further  analysis  and  decision  making. 
It  is  especially  important  to  provide  well-documented  data  in  a  timely  manner  to  those  who 
depend  on  readily  accessible,  accurate,  and  complete  data.  One  should  consider  the  following 
advice: 

•  Data  should  be  easily  discoverable  and  obtainable. 

•  Data  results  should  not  be  released  until  appropriate  authorities  have  granted  permissions. 

•  Complete  metadata  (characteristics  of  information-bearing  entities)  that  clearly  establish 
the  source  and  content  of  the  data  should  be  included. 

•  Data  should  be  properly  identified  and  protected  from  unauthorized  access  and 
inappropriate  use,  e.g.,  sensitive  data  in  reports,  metadata,  raw  and  manipulated  spatial 
and  non-spatial  data,  maps.  One  should  clearly  identify  classified  data  and  follow  all 
rules  of  use  and  distribution. 

At  times,  one  may  need  to  export  data  from  one  database  to  another,  e.g.,  from  Access  to  SPSS, 
ArcGIS,  or  FalconView  for  further  analysis.  Data  integrity  must  be  maintained  when 
transferring  data  between  software  tools.  One  can  most  easily  control  the  field  order,  i.e.,  order 
of  variables  in  the  resulting  'flat'  file,  while  the  data  is  still  in  the  relational  database.  One  should 
determine  field  types  early  on  in  the  database  development  process.  One  should  keep  data  type 
changes  to  a  minimum  in  order  to  minimize  risk  to  data  loss.  This  handbook  recommends  using 
the  American  Standard  Code  for  Information  Interchange  (ASCII)  text  for  its  nearly  universal 
readability  across  applications.  ASCII  text  typically  delimits  fields  by  commas,  tabs,  or  spaces, 
and  encloses  text  strings  by  single  or  double  quotes.  ASCII  text  requires  extra  steps  to  transfer 
data  between  applications  and  extra  care  regarding  data  formatting.  The  preferred  alternatives 
are  data  exports  directly  from  an  Access  .mdb  file  (such  as  ArcGIS)  or  link  to  an  .mdb  file  by 
way  of  a  database  connection  such  as  an  Object  Linking  and  Embedding  Database  (OLE  DB)  or 
an  Open  Database  Connectivity  (ODBC)  data  link. 

2.11  Data  Organization 


The  various  databases,  reports,  GIS  coverages,  etc.  create  a  large  number  of  files  and  folders  to 
manage.  Eor  example,  databases  are  occasionally  stored  in  two  versions  of  Access  in  order  to 
accommodate  data  users  with  different  software  versions.  Additionally,  GIS  data  are  sometimes 
stored  in  two  projections,  one  for  navigation  and  another  for  use  with  existing  base  GIS  data. 
Poor  file  management  leads  to  confusion  and  data  corruption. 

2.12  Data  Maintenance 


Data  sets  are  rarely  static.  They  often  change  through  additions,  corrections,  and  improvements 
following  the  archival  of  a  data  set.  There  are  three  main  caveats  to  this  process: 
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•  One  should  only  make  ehanges  that  improve  or  update  the  data  and  maintain  data 
integrity. 

•  When  archiving  the  data  set,  one  should  document  change  versions  made  to  the  data  set. 

•  One  should  prepare  backup  files  to  recover  from  mistakes  made  during  editing. 

2.13  Version  Control 


Secure  data  archiving  is  essential  for  protecting  data  files  from  corruption.  Version  control  of 
records  is  a  critical  piece  of  the  data  management  process.  Prior  to  making  any  major  changes  to 
the  dataset,  one  should  save  the  data  set  using  a  unique  version  number.  This  facilitates  tracking 
changes  over  time.  With  proper  controls  and  communication,  versioning  ensures  that  one  uses 
only  the  most  current  version  in  analysis.  One  should  use  a  logical  numbering  system  and  assign 
each  additional  version  a  sequentially  higher  number.  When  one  makes  new  versions  to  the  data 
set,  one  should  provide  all  frequent  users  with  the  most  recently  archived  version. 

2.14  Operational  Competence 


ORSA  analysts  must  remain  operationally  competent  across  the  full  spectrum  of  resident  skills. 
One  becomes  operationally  competent  by  integrating  oneself  into  the  staff  process  early  in  the 
planning  process  and  by  remaining  engaged  all  the  way  through  to  the  execution  of  operations. 
One's  ability  to  access  and  utilize  multiple  data  sources  ensures  that  the  commander  has  the  most 
accurate,  relevant,  and  timely  information  available.  For  example,  ORSA  analysts  deployed  with 
the  International  Security  Assistance  Force  (ISAF),  the  Combined  Joint  Special  Operations  Task 
Force  -  Afghanistan  (CJSOTF-A),  the  Joint  Special  Operations  Task  Force  -  Philippines 
(JSOTF-P),  and  the  Combined  Joint  Task  Force  -  Horn  of  Africa  (CJTF-HOA)  have  utilized 
multiple  data  sources  to 

•  analyze  the  projected  size  of  the  Afghan  National  Security  Forces  (ANSF); 

•  develop  a  process  to  locate,  validate,  store,  and  distribute  polling  data  for  analytical  use 
to  support  strategic  and  operational  issues  in  Operation  Enduring  Freedom  (OEF); 

•  analyze  poll  results  of  counterinsurgency  operations  to  gauge  the  success  of  efforts  to  win 
the  hearts  and  minds  of  the  local  population; 

•  develop  assessment  frameworks  to  inform  commanders  and  future  campaign  planners; 

•  consolidate  and  format  unusable,  individual  unit  reports  into  a  single  database  using  text 
extraction  techniques; 

•  assess  the  effectiveness  of  combat  and  security  operations  on  enemy  activities;  and 

•  establish  robust  database  systems  to  better  facilitate  analysis  of  assessments. 

2.15  Way  Ahead 


Data  for  decision  making  comes  from  a  variety  of  sources,  both  internal  and  external.  Because 
the  database  management  system  is  one  of  the  major  components  of  most  decision  management 
support  systems,  it  is  important  for  ORSA  analysts  to  be  familiar  with  the  latest  developments  in 
the  field.  Organizations  are  recognizing  that  their  data  contain  a  gold  mine  of  information. 
Consequently,  they  are  warehousing  and  mining  data  for  users  to  obtain  information  on  their 
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own  (through  a  variety  of  multidimensional  analytieal  tools  and  new  enterprise-wide  system 
arehiteetures)  and  establish  relationships  (through  data  extraetion  teehniques)  that  were 
previously  unknown.  New  emerging  tools,  sueh  as  Online  Analytieal  Proeessing  (OLAP), 
provide  on-the-fly  data  analysis  that  is  invaluable  to  analysts.  Moreover,  a  wide  variety  of  data 
formats  are  beeoming  available  through  innovative  networked  database  management  systems. 
One  ean  use  any  or  all  of  these  methods  to  enhanee  data  analysis  and  assist  eommanders  at  all 
levels  of  eommand  in  making  highly  informed  deeisions. 
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CHAPTER  3  DATA  ANALYSIS  TECHNIQUES 

3.1  Introduction 


Definition.  Data  analysis  is  the  act  of  transforming  and  modeling  data  to  extract  useful 
information  and  facilitate  conclusions  that  support  decision  making. 

Purpose,  This  chapter  highlights  several  good  data  analysis  techniques  used  by  previously 
deployed  ORSA  analysts.  As  a  deployed  ORSA  analyst,  one  makes  inferences  from  data  to 
recommend  courses  of  action  (CO As),  assess  specified  activities,  identify  trends,  etc. 

Data  analysis  is  an  objective  method  for  determining  whether  alternative  COAs  are  significantly 
different  from  each  other  and  for  providing  the  commander  with  resource  allocation 
recommendations.  Data  analysis  underpins  operational  assessments,  e.g.,  analysis  of  enemy  and 
friendly  activities  that  identify  changes  and  trends  in  the  operational  environment.  It  is  an  ORSA 
analyst's  responsibility  to  determine  which  data  analysis  technique  will  provide  commanders 
with  the  most  accurate  findings  to  assist  them  in  their  decision  making  process.  Commonly  used 
analytical  techniques  include  statistical  analysis,  simulation,  optimization,  network  flow  models, 
and  geospatial  analysis. 

3.2  Data  Cleaning 


Prior  to  beginning  analysis,  one  may  need  to  mine  and  clean  the  dataset.  Typically,  datasets 
contain  missing  data,  multiple  entries  of  the  same  event,  and  duplicate  events  entered  with 
different  terminology,  all  of  which  one  must  identify  and  correct  prior  to  analysis.  Furthermore, 
the  necessary  data  may  be  located  in  "text"  fields  that  require  one  to  mine  the  data  and  extract  the 
information.  In  a  deployed  environment,  "dirty  data"  are  quite  common.  ORSA  analysts  must 
not  use  "dirty  data"  in  their  analyses.  These  analyses  are  not  trivial;  they  often  become  part  of 
larger  studies.  Data  cleaning  is  a  labor-intensive  task. 

3.3  Statistical  Analysis 


Accurate  analyses  necessitate  selection  of  the  most  appropriate  technique  for  a  given  complex 
problem.  One  should  consider  events  that  may  have  an  impact  on  the  graphed  data,  e.g.,  major 
operations,  elections,  battlespace  transition,  and/or  significant  changes  in  the  political 
environment,  such  as  ratification  of  a  constitution/seating  of  a  parliament.  These  events  can 
explain  spikes  or  dips  in  the  data  and  place  the  data  in  context. 

Descriptive  Statistics,  This  handbook  recommends  visually  inspecting  the  data  set  using  a 
graph,  e.g.,  scatter-plot,  histogram,  or  time  series  chart  prior  to  conducting  the  analysis. 
Statistics  textbooks  provide  examples  of  descriptive  statistics,  e.g.,  Devore,  2012,  and  many  on¬ 
line  resources  outline  techniques  for  calculating  numerical  measures  and  developing  graphs. 

Statistical  Techniques,  One  can  find  the  most  relevant  trend- analysis  examples  by  referencing 
the  work  of  recently  deployed  ORSA  analysts.  Several  statistical  techniques  used  in  recent 
projects  conducted  in  theater  and  as  part  of  the  analytical  reachback  process  include  t-test; 
regression,  e.g.,  linear,  multiple,  and  logistic;  and  time  series  forecasting,  e.g.,  moving  average, 
weighted  moving  average,  and  Holt-Winters  method.  This  handbook  also  recommends 
"Forecasting:  Methods  and  Applications"  by  Makridakis,  Wheelright,  and  Hyndman  (1998). 
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3.4  Other  Analytical  Methods 


Deployed  ORSA  analysts  may  find  that  the  following  analytieal  methods  exeeed  eombat-theater 
resources,  i.e.,  time,  skill,  software;  however,  one  ean  still  use  these  methods  through  the 
reaehbaek  program. 

Forecasting,  Foreeasting  is  the  proeess  of  developing  estimates  for  future  outeomes  that  are 
likely  to  oeeur.  Modern  foreeasting  falls  into  one  of  two  broad  eategories:  eausal  foreeasting  or 
time  series  analysis.  Causal  foreeasting  prediets  how  an  uneertain  quantity  relates  to  other 
quantities.  Time  series  foreeasting  prediets  future  values  of  an  uneertain  quantity  based  on  past 
values  of  a  known  quantity.  Foreeasting,  i.e.,  predieting  future  events,  is  important  as  the  United 
States  eontinues  the  fight  against  terrorism.  Senior  leaders  may  ask  ORSA  analysts  to  develop 
tools  and  models  to  foreeast  enemy  aetivity,  both  temporally  and  spatially. 

One  eomprehensive  foreeasting  resouree  is  "Foreeasting  Prineiples:  Evidenee-based  Foreeasting" 
(2012),  available  from  http://www.foreoastingprinoiples.oom.  Makridakis,  Wheelwright,  and 
Hyndman's  "Foreeasting:  Methods  and  Applioations"  (1998)  is  also  a  good  resouree.  Makridakis 
et  al.  provide  desoriptions  and  examples  of  basio  foreeasting  tools,  time  series  deoomposition, 
simple  regression,  eto.  Additionally,  Sam  L.  Savage's  "Deoision  Making  with  Insight"  (2003) 
provides  good  explanations  of  eausal  and  time  series  foreeasting. 

Simulation.  Simulation  is  a  mathematioal  model  for  reereating  a  situation  repeatedly  to  more 
aeeurately  estimate  the  likelihood  of  various  outeomes.  One  ean  use  simulation  to  show  the 
eventual  real  effeets  of  alternative  eonditions  and  COAs,  Analysts  widely  use  Monte  Carlo 
eomputational  algorithms  for  simulating  the  behavior  of  various  physieal  and  mathematioal 
systems,  and  for  other  oomputations.  One  ean  use  simulation  in  foreeasting  as  well  (for  example, 
developing  reoommendations  for  resouree  allooation).  Albright  et  al.  (2011)  inolude  desoriptions 
and  examples  of  the  simulation  proeess.  Additionally,  Exoel  add-ins,  suoh  as  @Risk,  are 
designed  to  oonduot  simulations. 

Optimization.  One  ean  use  optimization,  i.e.,  math  programming,  to  minimize  or  maximize  a 
real  funotion  by  systematioally  ohoosing  the  values  of  real  or  integer  variables  from  within  an 
allowed  set.  One  ean  use  several  different  methods  to  oonduot  optimization,  to  inolude  linear 
programming,  network  flow  models,  and  oritioal  path  models.  Over  the  past  several  years, 
oommanders  and  senior  leaders  have  asked  ORSA  analysts  for  optimal  methods  of  employing 
teohnology,  transporting  teohnology  and  personnel  to  theater,  optimizing  ooverage  of  speoifio 
assets,  e.g.,  medioal  evaouation  [MEDEVAC]  and  intelligenoe,  surveillanoe,  and  reoonnaissanoe 
[ISR]  teohnology.  One  ean  oonduot  simple  optimization  problems  using  the  built-in  "Solver"  of 
Exoel.  Additional  Exoel  software  tools  are  available  for  sensitivity  analysis  to  determine  how 
the  optimal  solution  ohanges  as  one  or  more  input  variables  ohange.  Albright  et  al.  (2011) 
inolude  detailed  desoriptions  of  optimization  models  using  linear  programming. 

Sensitivity  Analysis,  Sensitivity  analysis  is  the  prooedure  for  determining  the  sensitivity  of 
alternative  outeomes  to  ohanges  in  one  or  more  parameters.  If  a  small  ohange  in  a  parameter 
results  in  a  relatively  large  ohange  in  the  outoome,  the  outoome  is  sensitive  to  that  parameter. 

This  may  mean  that  one  has  to  determine  the  parameter  very  aeeurately  or  redesign  the  outoome 
to  be  less  sensitive.  One  should  inoorporate  sensitivity  analysis  into  the  deoision  making  proeess 
to  determine  how  sensitive  the  reoommended  solution  is  to  various  parameters  of  the  model. 
Albright  et  al.  (2011)  provide  a  good  example  using  Exoel  to  oonduot  sensitivity  analysis. 


16  •  DATA  ANALYSIS  TECHNIQUES 


DAHB 


CAA-20 15094 


Additionally,  the  Precision  Tree  add-in  discussed  below  is  a  good  tool  for  conducting  this  type  of 
analysis. 

Gap  Analysis.  ORSA  analysts — more  often  than  not — have  only  incomplete  data  to  answer 
specific  questions.  Where  there  are  incomplete  data,  there  are  potential  gaps  in  the  analysis. 

One  should  identify  these  gaps  in  the  "limitations"  section  of  one's  final  written  report.  There  are 
several  ways  to  approach  missing  data,  to  include  (1)  deleting  incomplete  entries,  (2)  filling  in 
incomplete  entries  based  upon  the  most  similar  complete  entry  ("hot  deck  imputation"),  (3) 
simulation,  (4)  filling  in  complete  entries  with  the  sample  mean  ("mean  substitution"),  or  (5) 
using  a  learning  algorithm,  e.g.,  maximum  likelihood  to  infer  a  missing  entry.  In  one's  written 
reports,  one  should  annotate  the  methods  used  to  overcome  the  gaps  in  data.  As  additional  data 
become  available,  one  should  conduct  gap  analysis  to  "fill  in  the  blanks"  or  run  another  iteration 
of  the  analysis  to  determine  how  the  additional  data  change  the  results. 

3.5  Software  Utilization 


This  section  summarizes  software  packages  available  to  assist  in  the  execution  of  data  analysis. 
Senior  leaders  do  not  expect  ORSA  analysts  to  be  experts  in  all  of  them;  however,  they  do  expect 
ORSA  analysts  to  know  the  differences  between  them. 

Excel  built-in  Functions.  Several  built-in  functions  in  Excel  improve  the  efficiency  of  analysis. 
In  particular,  the  following  three  features  are  very  powerful:  pivot  tables,  sort  options,  and 
filtering.  These  features  allow  one  to  manipulate  and  visualize  data  quickly.  Moreover,  Excel 
has  "add-ins"  such  as  Analysis  ToolPak  for  statistical  analysis. 

Excel  add-in  Software.^  The  following  Excel  add-in  software  capabilities  are  available  for 
purchase: 

•  @Risk  for  risk  analysis.  With  this  feature,  one  can  define  probability  distributions  and 
run  simulations  for  input  and  output  variables  within  Excel. 

•  Precision  Tree  is  a  decision  analysis  add-in.  One  may  find  it  useful  in  optimization 
when  one  is  dealing  with  a  set  of  alternatives  such  as  decisions  on  using  new  weapons  or 
factoring  in  decisions  at  each  stage  of  contracting  and  development. 

•  Crystal  Ball  performs  Monte  Carlo  simulations  in  spreadsheets  that  automatically 
calculate  thousands  of  different  "what  if  scenarios  and  save  the  inputs  and  results  of 
each  calculation  as  an  individual  scenario.  Analysis  of  these  scenarios  reveals  the  range 
of  possible  outcomes,  the  probability  of  each  occurring,  which  inputs  have  the  greatest 
effect  on  the  model,  and  where  one  should  focus  one's  efforts. 

Macros.  One  can  create  a  Macro  to  automate  repetitive  actions,  i.e.,  a  series  of  recorded  actions. 
One  can  write  Macros  using  the  built-in  toolbar  or  Visual  Basic  for  Applications  (VBA)  code  in 
the  VBA  editor. 


^  Note:  No  endorsement  is  stated  or  implied  by  any  references  to  commercial  software 
applications  in  this  handbook. 
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3.6  Examples  of  Available  Software  Packages^ 


•  SPSS  is  used  for  statistical  analysis,  data  management,  and  data  documentation. 

•  Stata  is  used  for  data  management,  statistical  analysis,  graphics,  simulations,  and  custom 
programming. 

•  TRAC  creates  and  maintains  visualization  tools  for  DoD  analysts. 

•  DECMAT  is  a  well-known  multi-criteria  decision  matrix,  i.e.,  preference  matrix  used  in 
the  military  decision  making  process  (MDMP). 

•  JMP  is  visual  software  from  SAS  that  sets  itself  apart  by  linking  robust  statistics  with 
graphics  on  the  desktop,  producing  visual  representations  of  data  that  reveal  context  and 
insight  impossible  to  see  in  a  table  of  numbers.  JMP  allows  one  to  be  more  efficient, 
tackle  difficult  statistical  problems,  and  bring  one's  data  analysis  to  a  whole  new  level. 
Data  and  information  visualization,  design  of  experiments,  and  statistical  modeling 
techniques  from  simple  to  advanced  are  all  within  one's  grasp  with  this  powerful 
platform.  And  when  one  makes  JMP  one's  analytic  hub,  one  can  use  other  favorite  tools: 
SAS,  Excel,  and  R.^ 

•  MATLAB  is  used  for  algorithm  development,  data  analysis,  visualization,  and  numerical 
computation. 

•  Minitab  is  used  to  analyze  one's  data  and  improve  one's  products  and  services  with  the 
leading  statistical  software  used  to  implement  Six  Sigma  worldwide.® 

•  GAMS  is  used  for  mathematical  programming  and  optimization. 

•  R  (free  software)  is  used  for  statistical  computing  and  graphics. 

•  TABLEAU  helps  people  see  and  understand  data.  Tableau  helps  anyone  quickly 
analyze,  visualize,  and  share  information.'^ 


4  Note:  No  endorsement  is  stated  or  implied  by  any  references  to  commercial  software 
applications  in  this  handbook. 

5  JMP  software  is  available  from  http://www.jmp.com/en_us/software.html 

®  Minitab  software  is  available  from  http://www.minitab.com/en-US/Products 
Excerpt  from  the  TABEEAU  website:  http://www.tableau.com/ 
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3.7  Case  Study:  Time  Series  Analysis  in  a  Theatre  Strategic  Headquarters 

Time  Series  Analysis  in  a  Theatre  Strategic  Headquarter 

Marcus  Gaul  and  Eric  Jesse 

In:  Williams,  A.,  Bexfield,  J.,  Fitzgerald  Farina,  F.,  and  de  Nijs,  J.,  (Eds.).  Innovation  in 
Operations  Assessment:  Recent  Developments  in  Measuring  Results  in  Conflict  Environments. 
Innovation  in  Capability  Development  Series,  Vol.  1.  Norfolk,  VA:  NATO  Headquarters 
Supreme  Allied  Commander  Transformation  (2013). 

Abstract 

Time  series  analysis  in  its  many  facets  and  shapes,  from  data  exploration  to  spatial 
analysis,  and  from  qualitative  to  quantitative  considerations  offers  a  set  of  highly  valuable  tools 
for  analysts  in  support  of  senior  leader's  decision  making.  Time  series  analysis  can  not  only 
generate  crucial  insights  by  combining  time  and  spatial  domains,  but  it  can  help  to  identify 
unexpected  relationships  and  opportunities  that  can  be  exploited  by  a  command,  and  enable  the 
assessment  of  operational  effectiveness  and  developments  in  insurgent  tactics.  In  this  article  we 
present  the  fundamentals  of  time  series  analysis  and  discuss  some  of  its  applications  and  uses 
stemming  from  our  experiences  of  working  in  the  Afghan  Assessment  Group  (A AG)  at  the 
International  Security  Assistance  Force's  (ISAF)  Headquarters  in  Kabul,  Afghanistan. 

Introduction 

The  AAG  is  the  International  Security  Assistance  Force's  (ISAF)  Headquarters'  principal 
analysis  and  assessment  organization.  The  AAG  provides  regular  analysis  and  assessments  of 
the  campaign  to  the  ISAF  Commander,  the  Government  of  the  Islamic  Republic  of  Afghanistan 
(GIRoA),  and  the  governments  of  troop  contributing  nations,  the  press,  and  the  public.  In 
assessing  the  campaign,  AAG  uses  a  variety  of  analytic  methodologies,  one  of  which  is  time 
series  analysis,  the  focus  of  this  discussion.  Time  series  analysis,  in  particular,  leverages  the 
information  advantage  ISAF  has  achieved  through  its  advanced  theatre- wide  data  collection 
processes. 

Our  chapter  begins  with  a  general  review  of  time  series  analysis  and  associated 
terminology.  Then  we  will  cover  selected  common  analytical  issues  and  challenges,  to  include 

^  Used  by  permission  from  Marcus  Gaul  and  Eric  Jesse  In:  Williams,  A.,  Bexfield,  J.,  Fitzgerald 
Farina,  F.,  and  de  Nijs,  J.,  (Eds.).  Innovation  in  Operations  Assessment:  Recent  Developments  in 
Measuring  Results  in  Conflict  Environments .  Innovation  in  Capability  Development  Series,  Vol. 
1.  Norfolk,  VA:  NATO  Headquarters  Supreme  Allied  Commander  Transformation  (2013). 
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data  collection  and  consistency,  data  aggregation,  variable  selection,  seasonality,  and  non¬ 
constant  variance.  Aware  of  these  problems  that  frame  analytic  approaches,  we  next  discuss 
methods  AAG  employs  to  analyze  insurgent  activity  in  Afghanistan,  covering  year-over- year 
differencing,  seasonal  decomposition,  and  regression  analysis.  This  carries  into  a  discussion  on 
the  identification  of  statistically  significant  changes  in  insurgent  behavior.  Finally,  we  discuss 
the  incorporation  of  geospatial  considerations  with  temporal  data.  By  adding  spatial  data,  we  can 
visualize,  assess,  and  explain  trends,  which  might  otherwise  go  unobserved  by  analysts  and 
decision  makers.  We  conclude  with  a  discussion  of  the  challenges  quantitative  analysis 
encounters  during  the  ISAF  campaign,  and  the  facilitating  measures  taken  to  remove  them. 

Time  Series  Analysis 

A  time  series  is  a  collection  of  observations  obtained  through  repeated  measurements 
over  time.  Time  series  are  most  often  considered  subsets  of  stochastic  models  or  stochastic 
processes.  Stochastic  processes  concern  sequences  of  events  governed  by  probabilistic  laws. 
Stochastic  models  infer  characteristics  of  a  population  using  temporal  or  spatial  data.  Time 
series  analysis  utilizes  the  natural  temporal  order  of  the  data  to  extract  information  about  the 
underlying  stochastic  process. 

The  extent  of  assumptions  about  the  underlying  data  in  a  statistical  analysis  defines  two 
classes  of  statistics,  parametric  and  non-parametric.  The  most  common  models  utilize 
parametric  statistics.  Parametric  statistics  assume  that  the  data  have  some  underlying  structure, 
most  typically  about  the  underlying  distribution  (often  the  normal  distribution)  and  variance 
(often  constant  over  time).  Some  widely  known  parametric  models  are  ordinary  least  squares 
regression,  generalized  least  squares,  autoregressive  and  moving  average  models.  Non- 
parametric  (or  distribution  free)  models  do  not  assume  an  underlying  distribution  for  the  data; 
instead,  they  try  to  determine  the  underlying  system  structure  within  the  data.  As  a  result,  larger 
datasets  are  required  for  non-parametric  models  to  make  similar  conclusions  as  analogous 
parametric  test.  Non-parametric  methods  often  rely  on  ranking  or  ordering  of  data  for  analysis. 
The  Kolmogorov-Smirnov  test  is  one  such  well  known  non-parametric  method  used  to  test 
whether  a  data  set  follows  an  assumed  probability  distribution,  or  whether  two  data  sets  come 
from  the  same  generating  process. 

Our  discussion  of  time  series  analysis  focuses  on  parametric  statistics,  but  we  need  to  be 
aware  that  non-parametric  analysis  techniques  exist  when  parametric  assumptions  are  shown  to 
be  invalid. 
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Statistical  analysis  is  further  subdivided  into  linear  and  non-linear  analysis.  Linear 
analysis  assumes  the  dependent  variable  (response  variables)  can  be  explained  by  a  linear 
combination  of  independent  variables  (explanatory  variables).  Linear  analysis  assumes  an 
additive  relationship  between  variables  that  enables  a  closed  form  solution  to  estimation.  While 
linear  models  are  intuitively  easier  to  interpret  and  understand,  most  real  world  systems  do  not 
follow  strictly  additive  processes.  In  non-linear  analysis,  the  dependent  variable  is  explained  by 
a  non-linear  combination  of  independent  variables.  Examples  of  non-linear  functions  include 
Lorenz  curves,  exponential  functions,  logarithmic  functions,  and  Gaussian  functions. 

Exponential  and  logarithmic  functions  can  be  transformed  into  linear  forms  enabling  a  closed 
form  solution,  but  in  general,  non-linear  functions  use  numerical  techniques  to  estimate 
parameters. 

Time  series  analysis  can  be  used  to  answer  two  types  of  inquiries:  descriptive  and 
predictive.  Descriptive  inquiries  seek  to  ascertain  the  existence  of  hypothesized  relationships 
between  the  dependent  and  independent  variables.  Eor  example,  how  do  enemy-initiated  attacks, 
the  opportunities  for  leave,  seasonality,  or  major  Islamic  holidays  affect  attrition,  i.e.,  the 
planned  and  unplanned  loss  of  personnel  in  the  Afghan  National  Army?  Descriptive  analyses 
look  for  significance  and  direction  of  relationships  between  the  dependent  and  independent 
variables.  Predictive  analysis  projects  potential  future  system  states  based  on  the  historical 
relationship  between  a  dependent  variable  and  a  set  of  independent  variables.  An  example  is 
predicting  the  level  of  enemy  activity  in  2013  using  the  historical  relationship  between  the 

enemy  activity  level  and  independent  variables  such  as  the  number  of  coalition  forces^. 

Data  Collection  and  Aggregation 

ISAE  collects  vast  amounts  of  data  with  most  of  the  data  being  provided  by  military  units 
using  standard  reporting  procedures.  These  units  are  not  focused  primarily  on  data  collection,  so 
data  quality,  completeness,  and  accuracy  can  vary.  Reported  and  collected  data  also  tends  to 
evolve  with  mission  objectives,  potentially  adding  further  complexity  to  analysis.  The  collected 
data  are  refined  with  subsequent  reports  categorized  and  transmitted  to  an  analytical  database 
that  also  serves  as  a  historical  archive.  A  centralized  data  collation  function  provides  an 
additional  layer  of  quality  assurance  by  enhancing  data  consistency,  eliminating  duplicate 
reports,  and  identifying  and  reconciling  database  anomalies. 


9  See  (Gons,  2012)  for  an  example  of  such  a  scenario. 
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ISAF  tracks  additional  data  sources  beyond  military  operational  reporting.  These  reports  include 
open  source,  United  Nations,  and  non-governmental  organization  (NGO)  reporting.  Military 
operational  reporting  is  usually  entered  into  the  database  within  24  hours  of  occurrence  and  data 
entry.  This  lag  can  mislead  or  bias  near-term  analysis.  Understanding  the  data  is  the  foremost 
task  of  the  analysts,  since  quantity,  quality,  and  reliability  of  the  data  is  a  primary  factor  shaping 
the  analysis. 

Data  aggregation  is  the  next  major  consideration  for  the  analyst.  Data  should  not  be 
aggregated  beyond  the  relevant  level  of  inquiry.  For  example,  looking  at  theater-wide 
improvised  explosive  device  (lED)  events  would  be  an  inappropriate  means  of  assessing  a  new 
mine  rolling  system  in  Kandahar  Province. 

The  aggregation  of  security-related  data  in  Afghanistan  is  at  least  a  three-dimensional 
issue,  with  data  having  spatial,  temporal,  and  categorical  aspects.  Spatial  disaggregation  presents 
a  particular  challenge  due  to  the  number  of  ways  Afghanistan  can  be  divided.  ISAF  seeks  to 
map  data  with  an  accuracy  of  a  few  meters  to  a  specific  military  grid  reference  system  (MGRS) 
coordinate,  but  this  level  of  accuracy  is  not  always  achieved,  especially  for  reports  from  the 
Afghan  National  Security  Force  (ANSF).  Almost  all  data  are  identified  at  the  district  level.  The 
approximately  400  districts  are  in  34  provinces,  which  in  turn  are  part  of  6  geographically 
distinct  regions  in  Afghanistan:  East,  South,  Southwest,  West,  North,  and  the  Capital  -  the  ISAE 
Regional  Commands  (RCs).  Numerous  further  groupings  are  possible,  for  example  defined  by 
climate,  terrain,  and  population  demographics. 

Aggregation  can  influence  analytic  results.  Simpson's  paradox  is  the  phenomenon  where 
a  trend  that  appears  in  data  subsets  disappears  or  reverses  when  these  subsets  are  combined.  As 
a  hypothetical  example,  we  might  find  an  improving  security  situation  in  28  of  Afghanistan's  34 
provinces  when  analyzed  separately,  but  when  looking  across  the  entire  theater,  the  trend  could 
be  reversed  and  security  would  appear  to  be  worsening.  This  trend  could  be  driven  by 
dominating  negative  conditions  in  just  two  provinces,  for  example  Helmand  and  Kandahar. 

Disaggregation  can  potentially  obscure  important  trends  due  to  the  effect  of  specific 
events,  operations,  or  simple  random  noise.  Disaggregation  can  also  limit  the  power  of  statistical 
analysis  by  reducing  the  number  of  observations.  There  is  no  "correct"  level  of  aggregation  to 
employ  across  all  analysis.  Aggregate  models  are  very  important  to  understanding  general 
theater  trends,  while  disaggregated  models  enable  the  understandings  that  are  important  at 
operational  and  tactical  levels.  Disaggregate  models  help  with  the  analysis  of  local  effects  of 
actions  such  as  clear/hold  operations,  cache  finds,  or  weapon  seizures.  The  appropriate  level  of 
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aggregation  is  a  judgment  call  by  the  operational  analysts  based  on  their  understanding  of  the 
question  and  the  tractability  of  the  data. 

Adjustments  for  Seasonality 

Insurgent  activity  in  Afghanistan  follows  a  strong  cyclical  process.  This  type  of  recurring 

pattern  in  data  is  known  as  seasonality.^^  Nationwide  insurgent  activity  in  Afghanistan  is 
closely  tied  to  the  rhythm  of  daily  life,  with  the  progression  of  the  seasons  strongly  influencing 
engagement  levels.  Planting  and  harvesting  crops  demand  the  attention  of  individuals  who  might 
otherwise  be  fighting  coalition  or  Government  of  the  Islamic  Republic  of  Afghanistan  (GIRoA) 
forces,  while  the  winter  cold  and  snows  close  mountain  passes  limits  movement  that  results  in 
diminishing  contact  between  insurgents  and  coalition  forces. 

As  measures  for  insurgent  activity,  we  consider,  for  example,  security  incidents^^  or 
enemy-initiated  attacks^^  (EIAs).  EIAs  provide  a  perspective  on  the  active,  executed 
components  of  insurgent  engagements  against  ISAE  and  the  ANSE,  while  security  incidents  also 
include  elements  that  failed  to  execute  such  as  explosive  devices  that  were  found  and  cleared. 

Eor  the  purpose  of  this  article,  we  consider  enemy-initiated  attacks  as  an  approximation  for 
insurgent  activity. 

Eigure  2  shows  the  periodic  pattern  of  insurgent  activity  over  the  period  from  1  January 
2007  through  31  December  2012.  In  July  of  2011,  EIAs  averaged  over  700  per  week.  Eive 
months  later,  in  December  2011,  EIAs  averaged  just  over  350  per  week,  a  drop  of  approximately 
50%.  Interpreting  this  decrease  as  a  sign  of  an  improving  security  situation  could  be  misleading 
for  several  reasons.  Eirst  of  all,  this  comparison  fails  to  account  for  seasonality,  i.e.,  fluctuations 
in  the  insurgent  activity  due  to  the  seasonal  temperature  changes  over  the  course  of  a  year.  We 
can  see  this  by  considering  EIAs  several  months  later.  In  July  of  2012,  EIAs  again  averaged 
over  700  per  week. 


'0  Many  time  series  show  cyclic  variations  that  are  known  as  seasonal  fluctuations  or  seasonality. 
Insurgent  activity  in  large  parts  of  Afghanistan  is  correlated  to  the  seasonal  temperature 
fluctuations  over  the  course  of  a  year. 

Security  incidents  comprise  enemy-initiated  attacks  (enemy-initiated  direct  fire,  indirect  fire, 
surface-to-air  fire  and  executed  lED  attacks),  as  well  as  potential  lED  attacks  such  as  lEDs  and 
mines  that  were  found  and  cleared. 

12  Enemy-initiated  attacks  is  a  term  that  is  used  for  the  total  of  enemy-initiated  direct  fire, 
indirect  fire,  and  surface-to-air  fire  attacks,  as  well  as  executed  lED  attacks,  namely  lED 
explosions  and  mine  strikes. 
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Figure  2.  Weekly  enemy-initiated  attacks  in  Afghanistan  (1  January  2007-  31  December 

2012). 

An  assessment  of  the  security  situation  should  include  not  only  insurgent  activity,  but 
also  quantitative  and  qualitative  metrics  of  population  perceptions  of  development  and 
governance,  as  well  as  interdependencies  among  the  variables.  A  comprehensive  analysis  of  this 
type  extends  beyond  the  scope  of  this  chapter. 

When  analyzing  the  insurgency,  it  is  useful  to  differentiate  between  systemic  changes 
and  those  changes  that  are  attributable  to  the  cyclical  pattern  of  the  conflict.  ISAF  assessments 
regularly  utilize  three  approaches  to  adjust  for  seasonality  year-over-year  differences,  classical 
seasonal  decomposition,  and  regression  analysis. 

Year-over-year  (YoY)  differences  allow  comparisons  between  seasonally  analogous 
periods,  e.g.,  comparing  EIAs  in  January  2013  with  January  2012.  The  comparison  window  is 
important,  as  periods  that  are  too  small  risk  confusing  natural  variance  in  the  data  with 
significant  trends.  HQ  ISAF  AAG  found  that  a  1-week  period  was  the  shortest  possible  period  to 
identify  meaningful  changes  in  insurgent  activity  from  data  noise.  On  the  other  hand,  if  YoY 
analysis  spans  too  much  time,  operationally  relevant  factors  may  be  obscured,  giving  the 
commander  misleading  information. 

When  seasonal  trends  evolve  slowly,  it  is  sometimes  possible  to  compare  the  last  period 
with  that  immediately  preceding  to  search  for  changes,  e.g.,  for  data  with  an  annual  seasonal 
trend  we  can  compare  the  last  week  with  the  preceding  week.  This  is  an  analogous  approach  to 
more  traditional  YoY  differencing  in  that  the  two  periods  should  have  very  similar  seasonal 
influences.  When  they  differ  considerably,  it  could  be  indications  of  an  emerging  trend. 
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A  second  approach  commonly  employed  to  seasonally  adjust  the  data  is  classical 
seasonal  decomposition.  This  method  removes  seasonal  fluctuations  from  the  data  allowing  the 
identification  of  long-term  trends.  The  technique  factors  time  series  data  (Xt)  into  two  parts:  a 
seasonal  component  (St)  and  a  trend  component  (Tt).  While  both  additive  and  multiplicative 
decompositions  are  available,  multiplicative  decompositions  perform  best  when  modeling 
insurgent  activity  in  Afghanistan  (Xt  =  Tt  *  St).  Since  we  know  a  priori  that  the  cycle  affecting 
the  time  series  of  enemy-initiated  attack  data  (Xt)  is  annual,  we  can  calculate  the  two 
components  of  the  seasonal  decomposition  using  the  following  steps: 

1)  Compute  a  moving  average  for  the  series  Xt,  with  the  moving  average  window  being 
of  equal  length  to  one  season  (i.e.,  12  months,  52  weeks,  365  days).  This  step 
removes  intra-seasonal  variability  from  the  data  resulting  in  a  smoothed  time  series 
(SMt). 

2)  Compute  the  ratio  of  the  observed  (Xt)  and  smoothed  (SMt)  series.  This  ratio  is  the 
raw  seasonal  component. 

3)  Next  the  average  seasonal  component  (St)  is  computed  by  averaging  the  raw  seasonal 
components  for  each  point  in  the  time  series  (i.e.,  each  month,  each  week,  and  each 
day).  Various  forms  of  averaging  the  raw  seasonal  components  to  determine  the 
average  seasonal  component  are  possible.  We  recommend  normalized  medial 
averaging.  This  approach  excludes  outliers  from  the  average,  preventing  particularly 
abnormal  observations  from  influencing  the  seasonal  adjustment  in  the  same  period 
for  other  cycles. 

4)  The  original  time  series  (Xt)  can  now  be  seasonally  adjusted  by  dividing  it  by  the 
average  seasonal  component  (St),  producing  the  trend  component  (Tt). 

A  seasonal  decomposition  of  weekly  EIAs  in  Afghanistan  is  depicted  in  Figure  3  showing  the 
usefulness  of  this  approach. 
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Figure  3,  Seasonal  decomposition  of  weekly  enemy-initiated  attacks  in  Afghanistan  (1 

January  2007-  31  December  2012), 

Regression  analysis  is  the  third  approach  to  seasonally  adjust  data  and  identify  trends. 
Regression  analysis  is  used  to  estimate  relationships  between  a  dependent  variable  and  one  or 
more  independent  variables.  A  major  strength  of  regression  analysis  is  that  it  assesses  the 
relationships  between  multiple  independent  variables  to  the  dependent  variable.  Variable 
selection  is  the  process  of  identifying  control  variables  in  regression  analysis.  In  Afghanistan, 
factors  known  to  influence  insurgent  activity  include  coalition  and  ANSF  force  levels  and 
significant  events  such  as  elections  and  Islamic  holy  days.  The  number  of  insurgent  attacks  on 
ISAF  and  ANSF  patrols  is  related  to  the  location  and  frequency  of  these  patrols.  In  the  case  of 
elections  and  Islamic  holy  days,  insurgent  activity  has  two  opposing  characteristics.  Prior  to  and 
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during  elections,  insurgents  attempting  to  intimidate  the  population  accelerate  attacks. 
Conversely,  some  Islamic  holy  days  see  lower  levels  of  activity  due  to  religious  customs. 

When  selecting  control  variables  for  factors  that  influence  insurgent  activity,  it  is 
important  to  distinguish  between  regular  periodicity  and  irregular  periodicity,  i.e.,  periods  or 
events  that  recur  at  the  same  time  each  year  and  those  periods  or  events  that  occur  on  different 
calendar  dates.  Events  that  occur  at  the  same  time  annually  are  subsumed  in  the  seasonal 
adjustment.  For  example,  insurgent  activity  decreases  during  the  poppy  harvest  season  each 
year,  but  since  the  harvest  occurs  at  approximately  the  same  time  each  year  it  is  captured  by  the 
seasonal  adjustment.  Islamic  holy  days,  however,  follow  a  lunar  calendar  in  which  a  year  is 
always  12  lunar  months,  making  the  months  not  linked  to  the  seasons.  Key  dates  shift  each  year 
10  to  12  days.  As  a  result,  the  effects  of  the  annual  Islamic  observances  of  Fid  al-Fitr  (the 
"Festival  of  Fast-Breaking"  marking  the  end  of  Ramadan)  and  Eid  al-Adha  (the  "Festival  of 
Sacrifice")  will  not  be  accounted  for  in  a  seasonal  adjustment.  Regression  models,  being  the 
most  powerful  of  the  seasonal  adjustment  techniques  will  be  explored  in  greater  detail  below. 

Linear  Regression  Models 

There  are  several  types  of  regression  models.  In  this  section  we  discuss  how  simple 
linear  regression  models  can  be  applied  to  estimating  trends  in  enemy-initiated  attacks.  Other 
approaches  will  be  addressed  in  later  sections.  Determining  the  best  model  structure,  as 
measured  by  statistically  accurate  parameter  estimates,  can  be  challenging.  Ordinary  least 
squares  (OFS)  regression  assumes  the  independent  variables  are  additive  (linear). 

In  Afghanistan,  as  discussed  earlier,  some  of  the  major  factors  related  to  insurgent 
activity,  in  particular  seasonality,  have  a  multiplicative  relationship.  We  use  a  log  transformation 

converting  an  approximately  multiplicative  process  into  a  linear  model.  ^3  Techniques  are 
available  for  treating  seasonal  adjustments  simultaneously  with  other  explanatory  variables.  If 

Coalition  force  levels  strongly  influence  insurgent  activity  levels,  but  coalition  force  levels 
have  varied  significantly  from  less  than  40,000  troops  in  2006  to  over  150,000  at  the  peak  of 
the  surge  in  20  IF  Asa  result,  the  variance  in  insurgent  activity  has  changed  significantly  over 
time.  Uneven  variance  over  time,  heteroskedasticity,  causes  ordinary  least  squares  estimation 
to  be  inefficient  and  overstates  statistical  confidence  in  measured  estimates.  A  log 
transformation  of  the  dependent  variable  makes  a  multiplicative  model  additive  and  in  many 
cases  significantly  reduces  or  entirely  removes  the  problem  of  heteroskedasticity.  There  are 
numerous  methods  to  stabilize  variance.  A  square  root  transformation  is  often  also  appropriate 
for  count  processes.  Other  variance  stabilizations  include  the  Anscombe  transform 
(Anscombe,  1948),  Freeman-Tukey  transform  (Freeman  and  Tukey,  1950),  and  Box-Cox 
transform  (Box  and  Cox,  1964)  amongst  others. 
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treated  separately,  seasonal  adjustments  will  include  some  of  the  effects  of  the  explanatory 
variables.  Treating  the  adjustment  simultaneously  with  other  explanatory  factors  ensures  that  the 
seasonal  adjustment  itself  does  not  include  effects  that  are  more  correctly  attributed  to  the  other 
explanatory  factors  in  the  model.  In  a  regression  model,  the  addition  of  "indicator"  variables  for 
each  observation  period  in  the  seasonal  cycle  (which  is  1  year  for  the  Afghan  insurgency)  is 
equivalent  to  a  classical  seasonal  decomposition.  An  indicator  variable  is  assigned  a  value  of 
one  when  a  condition  is  present  and  zero  otherwise.  For  example,  the  seasonal  decomposition  in 
a  regression  model  using  weekly  data  includes  51  additional  indicator  variables,  one  for  each 
week.  The  52nd  week  is  captured  in  the  intercept,  as  it  is  perfectly  identified  by  the  5 1  other 
weekly  indicator  variables  and  therefore  cannot  be  directly  estimated. 

A  simple  (multiple)  linear  regression  model  of  weekly  insurgent  activity  expressed  by 
seasonality,  coalition  and  Afghan  Nation  Army  (ANA)  force  levels,  and  the  holy  days  of  Bid  al- 
Fitr,  and  Eid  al-Adha,  can  be  written  as  follows  (with  each  variable  considered  a  vector  of  the 
data  indexed  by  week): 


where 


Y  =  log  (insurgent  activity)  , 

=  log  (coalition  forces)  , 

X2  =  log  (ANA  forces)  , 

X3  =  0(Eid  al-Fitr),  where  O  is  a  scalar  function  assigning  the  value  of  one  to  a  week  in 
which  the  days  of  Eid  al-Fitr  fall,  and  the  value  of  zero  otherwise, 

X4  =  0(Eid  al-Adha),  where  O  is  a  scalar  function  assigning  the  value  of  one  to  a  week 
in  which  the  days  of  Eid  al-Adha  fall,  and  the  value  of  zero  otherwise, 

X5  =  O (elections),  where  O  is  a  scalar  function  assigning  the  value  of 

one  to  a  week  in  which  an  election  falls,  and  the  value  zero  otherwise, 

^6. ..57  =  ®(week)  is  the  indicator  (or  dummy)  variables  for  each  observation  period,  i.e., 
week,  in  the  year,  and 
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£  is  the  residual  error. 

The  parameters  P  are  computed  using  OLS,  i.e.,  by  minimizing  the  sum  of  squared 
deviations  between  the  dependent  variable  (observations)  and  the  estimate  using  the  independent 
variables.  Each  parameter  measures  the  observed  historical  relationship  the  independent  variable 
has  with  the  dependent  variable. 

The  observed  and  predicted  results  of  this  model  for  the  Afghan  theater  of  operation  are 
shown  in  Figure  4.  The  results  of  the  OLS  for  nationwide  and  four  Regional  Command  (RC) 
models  are  presented  in  Table  1. 
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Figure  4,  Observed  versus  predicted  enemy-initiated  attacks  (EIAs), 


Coefficient 

Theater 

RC  South 

RC 

Southwest 

RC  East 

RC  North 

-9.76  (0.26) 

-9.31  (0.46) 

-15.59  (0.54) 

-8.75  (0.29) 

-15.59  (0.54) 

Intercept 

log(Coalition 

1.11  (0.07) 

0.76  (0.13) 

1.98  (0.15) 

0.48  (0.08) 

1.98  (0.15) 

) 

*** 

*** 

0.21  (0.06) 

0.38(0.11) 

0.63  (0.07) 

log(ANA) 

-0.22  (0.12)  * 

-0.22  (0.12)  * 

-0.26  (0.07) 

-0.30  (0.12) 

-0.33  (0.14) 

-0.21  (0.08) 

-0.33  (0.14) 

Bid  al-Fitr 

** 

** 

** 

Bid  al-Adha 

-0.10(0.07) 

-0.18  (0.12) 

-0.04  (0.14) 

-0.04  (0.08) 

-0.04  (0.14) 

Election 

0.54  (0.11) 

0.50  (0.19) 

0.63  (0.12) 

Week 

0.36  (0.22) 

0.36  (0.22) 

0.9525 

0.8596 

0.8736 

0.9423 

0.8736 

Table  Format:  Coefficient  (Standard  Error)  Significance 
Significance:  ***  at  0.01,  **  at  0.05,  *  at  0.1 


Table  1,  Results  of  the  OLS  model  by  region  in  Afghanistan, 
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We  see  in  Table  1  that  the  parameters  point  in  the  hypothesized  directions.  Insurgent  activity 
rises  with  increases  in  coalition  and  ANA  force  levels,  Islamic  holidays  lead  to  decreases  in 
insurgent  activity,  and  elections  result  in  increased  insurgent  activity.  The  only  variable  that 
does  not  point  in  its  hypothesized  direction  is  ANA  troop  level  in  RC  North.  This  may  have 
occurred  because  the  ANA  troop  levels  are  modeled  at  the  theater  level,  and  this  may  be  an 
inadequate  measure  of  ANA  forces  in  RC  North.  It  could  also  have  occurred  by  chance  with  the 
measured  effect  being  only  weakly  significant  at  a  confidence  level  of  0.1,  i.e.,  with  a  90% 
chance  of  being  a  significant  relationship. 

Looking  at  the  nationwide  model,  assuming  all  else  to  be  equal,  the  interpretation  of  the 
model  results  is  as  follows: 

•  A  10%  decrease  in  the  number  of  coalition  forces  is  correlated  with  a  1 1%  decrease  in 
insurgent  activity 

•  A  10%  increase  in  ANA  forces  is  correlated  with  a  2%  increase  in  insurgent  activity 

•  A  week  encompassing  Eid  al-Fitr  has  a  26%  decrease  in  insurgent  activity 

•  An  election  weeks  results  in  a  54%  increase  in  insurgent  activity 

Finally,  it  should  be  noted  that  each  of  the  five  regression  results  in  Table  1  have  an 
associated  R  .  This  is  sometimes  called  the  coefficient  of  determinations  and  it  is  a  measure  of 
the  proportionate  reduction  of  total  variation  associated  with  the  use  of  the  independent 
variables.  It  can  take  on  values  between  zero  and  one,  with  higher  numbers  like  those  seen  in  the 
table  implying  that  the  regression  model  is  a  good  fit  to  the  data. 

The  levels  of  significance  displayed  in  Table  1  are  calculated  assuming  that  the  residuals 
are  normally  distributed,  with  constant  variance,  and  are  not  correlated  with  each  other.  In 
general,  it  is  necessary  to  test  these  implicit  assumptions  of  parametric  models  and  verify  model 
validity.  Violation  of  these  assumptions  can  bias  results  and  mislead  conclusions. 

Autocorrelation  is  one  common  issue  in  time  series  data.  It  occurs  when  the  current 
observation  is  correlated  with  an  observation  at  another  time  period,  e.g.,  the  immediately 
preceding  one,  violating  the  independence  assumption  of  OFS. 

Non-constant  variance  is  a  second  potential  issue.  As  coalition  force  levels  have 
varied  significantly,  from  less  than  40,000  troops  in  2006  to  over  150,000  at  the  peak  of  the 
surge  in  201 1,  the  variance  in  reported  insurgent  activity  has  changed  significantly  over  time. 
Uneven  variance  over  time,  heteroskedasticity,  may  result  in  overstated  statistical  confidence  in 
parameter  estimates. 
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We  can  test  the  OLS  assumptions  visually  and  with  more  formal  tests.  Figure  5  shows 
selected  residual  analysis  for  the  nationwide  model  in  Table  1.  The  top  plot  shows  studentized 
residuals.  Studentized  residuals  are  normalized  by  their  variance  and  thus  should  align  with  a 
standard  normal  curve  with  most  observations  falling  between  -2  and  +2.  The  bottom  plot 
compares  a  histogram  of  the  residuals  to  a  normal  distribution.  We  can  see  that  the  residuals  do 
not  appear  to  violate  the  normality  assumption. 

The  residuals  show  signs  of  autocorrelation,  with  insurgent  activity  in  one  period 
appearing  strongly  related  to  insurgent  activity  in  adjacent  periods,  as  visible  in  Figure  5. 

Having  homoskedastic  residuals  (residuals  with  constant  variance  over  time)  that  are  reasonably 
balanced  (centered  about  zero  over  time),  our  estimates  are  unbiased,  but  our  confidence  in  these 
estimates  is  overstated.  In  order  to  correct  the  effects  of  autocorrelation  (or  heteroskedasticity  if 
balanced)  in  the  error  terms  from  OLS  regression,  the  autocorrelation-consistent  Newey-West 
variance-covariance  matrix  estimator  of  the  residuals  is  often  used.  Alternatively,  we  can  utilize 
a  generalized  linear  model  and  explicitly  address  the  autocorrelation. 
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Figure  5.  Selected  residual  analyses. 

Autoregressive,  Integrated  Moving  Average  (ARIMA)  Models 

Ordinary  least  squares  is  one  of  many  forms  of  regression  analysis  (Hamilton,  1994).  As 
the  event  data  are  actually  count  data,  Poisson  or  negative  binomial  models  under  certain 
conditions  can  yield  much  more  accurate  predictions.  If  data  fidelity  improved  and  limitations  in 
computing  power  were  removed,  point  process  models  like  a  Cox  process  result  in  even  higher 
fidelity  characterizations  of  the  data  over  space  and  time  (Cox  1955;  Cox  and  Isham,  1980;  Cook 
and  Lawless,  2007;  Allen  et  ah,  2008). 

A  class  of  models  that  perform  well  in  forecasting  non-stationary  time  series  data,  like 
insurgent  activity  in  Afghanistan,  are  autoregressive,  integrated  moving  average  (ARIMA) 
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models  (Box  et  al.  2008)^4  ARIMA  models  consist  of  three  parts,  an  autoregressive  (AR)  part, 
an  integrated  (I),  and  a  moving  average  part  (MA).  The  model  is  usually  referred  to  as  an 
ARIMA(p,  d,  q)  model  where  the  parameters  p,  d,  and  q  are  non-negative  integers.  The 
parameter  p  refers  to  the  order  of  the  autoregressive  part,  i.e.,  the  number  of  autoregressive 
terms,  d  refers  to  the  order  of  the  involved  differencing  (note  that  a  time  series  that  needs  to  be 
differenced  to  be  made  stationary  is  said  to  be  an  "integrated"  version  of  a  stationary  series),  and 
q  is  the  order  of  the  moving  average  part  of  the  model.  An  ARIMA  model  specification  can  be  a 
subset  of  the  autoregressive,  integrated,  and  moving  average  components.  None  of  the  parts  of 
the  ARIMA  is  required  be  included  in  its  final  specification. 

An  autoregressive  model  seeks  to  define  the  current  observation  as  a  function  of 
preceding  observations.  For  example,  an  autoregressive  model  of  order  one  is  given  by 
Jt  —  Po  ^  PiVt-i  +  while  the  general  autoregressive  model  of  order  p  is  given  by  = 

Po  +  Ta=iPiyt-i  + 

The  integrated  part  of  the  model  utilizes  successive  first  differences.  For  example,  for  a 
difference  of  order  2,  the  relationship  analyzed  is  with  the  variable  defined  by  the  difference 
between  the  current  and  previous  periods  minus  the  difference  between  the  previous  period  and 
the  period  preceding,  mathematically  expressed  as  =  (x^  —  Xt_i)  —  (xt_i  —  Xt_2).  The  most 
common  use  of  such  an  integrated  term  is  to  account  for  a  quadratic  trend  in  the  data. 

A  moving  average  model  defines  the  current  observation  as  the  mean  p  of  the  series,  and 
current  and  previous  error  terms  £f  For  example,  a  moving  average  model  of  order  one  is  given 
by  Tt  =  F  +  +  Pi^t-i^  and  a  moving  average  model  of  order  q  is  given  by  y^  =  p  -I- 

Pi^t-i-  Because  the  lagged  error  terms  are  not  directly  observable,  iterative  non-linear 
fitting  procedures  need  to  be  used  in  place  of  linear  least  squares  to  estimate  model  parameters. 

A  seasonal  ARIMA  is  an  adaptation  of  the  standard  ARIMA  that  incorporates  a  priori 
information  about  the  recurring  seasonal  pattern.  In  a  seasonal  ARIMA  model  with  a  recurrence 
period  of  time  S,  observation  y^  is  predicted  by  data  values,  differences,  and  errors  with  time 
lags  that  are  multiples  of  S  (the  span  of  the  complete  seasonal  cycle).  For  example,  a  seasonal 
autoregressive  model  of  order  two  is  given  by  y^  =  Pq  +  PiYt-s  +  PiJt-is  + 


'4  A  stationary  process  is  a  stochastic  process  whose  joint  probability  distribution  does  not 
change  over  time.  This  means  parameters  like  the  mean  and  variances  are  constant  over  time, 
i.e.,  time  invariant.  The  data  here  exhibits  cyclostationary  properties  -  it  varies  annually.  In 
order  to  perform  time  series  analysis  we  transform  the  data  to  become  stationary  (Priestley 
1988). 
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An  ARIMA  model  as  briefly  discussed  above  has  several  benefits.  It  allows  us  to  model 
seasonality  using  fewer  terms  than  in  the  (multiple)  linear  regression  model  described  earlier, 
increasing  estimation  power.  It  also  allows  simultaneous  estimation  of  non- seasonal  factors  like 
force  levels,  or  Islamic  holy  days  Bid  al-Adha  and  Bid  al-Bitr.  An  ARIMA  model  will  also 
identify,  capture,  and  reflect  unspecified  underlying  processes  that  might  be  eliminated  as  noise 
in  other  types  of  estimation  models,  thus  making  it  a  very  powerful  tool  for  forecasting  and 
prediction. 

At  the  theater  strategic  level  we  have  used  seasonal  ARIMA  models  to  forecast  insurgent 
activity  taking  several  factors  into  account,  primarily  the  projected  troop  level  changes.  Bigure  6 
shows  the  model  predictions  of  insurgent  activity  in  2012  based  on  historical  data  from  2007  to 
2011.  The  troop  level  changes  in  2012  that  are  factored  in  the  prediction  are  shown,  as  well  as 
the  confidence  bounds  on  the  predicted  BIA  levels.  The  seasonal  ARIMA  has  a  non-seasonal 
part  of  (1,  0,  0)  and  a  seasonal  specification  of  (0,  2,  1);  it  also  includes  control  variables  for 
troop  levels.  Bid  al-Bitr,  and  Bid  al-Adha;  however,  no  examination  of  the  structural,  systemic 
causes  for  insurgent  activity,  or  the  changes  and  dynamics  are  considered. 
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Prediction  of  Enemy-Initiated  Attacks  for  2012 


Observed  ElAs  Prediction  - Upper/Lower 85%  Conf.  Bound 
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Figure  6,  Seasonal  ARIMA  prediction  model  for  insurgent  activity. 

Note  that  the  model's  overestimation  of  insurgent  activity  for  201 1,  as  seen  in  Figure  6,  is  caused 
by  not  controlling  for  voter  registration  periods  or  the  18  September  2010  Parliamentary 
Election. 


Identifying  Operationally  Relevant  Changes 

The  identification  of  operationally  relevant  changes  is  one  goal  of  analysis  at  a  strategic 
headquarters.  The  ability  to  rapidly  evaluate  and  assess  changes,  identify  opportunities  first,  and 
provide  the  option  to  react  faster,  can  provide  a  significant  advantage  over  the  opposition.  ISAF 
developed  a  methodology  that  utilizes  regression  models  to  quickly  identify  measurable  changes 
in  insurgent  activity  and  determine  the  statistical  significance  of  this  change  (Jesse,  2013). 

The  procedure  begins  by  defining  a  model  to  explain  insurgent  activity,  and  estimating  it 
against  a  benchmark  period  of  data.  We  next  compute  the  residuals  for  this  benchmark  period, 
which  are  defined  as  the  difference  between  model  predictions  and  observed  insurgent  activity. 
The  residuals,  or  fitting  errors,  to  regression  models  often  follow  a  distribution  that  is,  in  many 
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cases,  a  normal  distribution  with  mean  zero.  We  can  estimate  the  parameters  of  this  residual 
distribution  and  design  a  statistical  test  to  search  for  un-modeled  changes  in  insurgent  activity. 
With  the  model  predictions  over  the  benchmark  period,  we  can  forecast  insurgent  activity  levels 
for  new  time  periods  and  compare  these  with  observed  insurgent  activity.  From  the  sequence  of 
errors  obtained  by  this  procedure,  and  the  known  information  about  the  residuals'  distribution, 
we  can  find  the  statistical  likelihood  for  a  significant  change  (Chandler  and  Scott,  2011). 

One  use  of  the  above  approach  at  ISAF  is  to  monitor  daily  insurgent  activity.  We  use  a 

two-stage  least-squares  model  of  daily  enemy-initiated  attacks  baselined  on  the  period  from  1 
January  2010  to  31  October  2012  to  monitor  changes  in  insurgent  activity. 

The  advantage  of  a  two-stage  model  is  its  separate  accounting  for  additive  and 
multiplicative  relations.  The  first  stage  models  enemy-initiated  attacks  using  ISAF  and  ANSF 
troop  levels,  which  explains  much  of  the  growth  in  insurgent  activity.  For  the  second  stage,  the 
observed  enemy-initiated  attack  values  are  normalized  by  the  predictions  from  the  first  stage. 
This  procedure  results  in  the  dependent  variable  being  normalized  around  one  (seasonal 
fluctuations  persist),  and  it  should  remove  any  heteroskedasticity.  Fitting  the  second  stage  on  the 
normalized  dependent  variable,  independent  variable  coefficients  estimate  the  relative  change 
from  the  mean  caused  by  the  variables.  In  the  example,  fitting  the  second  stage  model,  we  get 
estimates  of  the  effects  of  the  Islamic  holidays  of  Ramadan,  Fid  al-Adha,  and  Bid  al-Fitr 
seasonality  approximated  by  month. 

In  the  next  step,  we  compute  the  residuals  of  the  two-stage  model.  The  residuals  of  this 
model  should  closely  approximate  a  normal  distribution  with  a  mean  of  zero  and  constant 
standard  error.  We  then  employ  a  Mest  to  calculate  the  likelihood  that  the  observed  insurgent 
activity  deviates  significantly  from  activity  explained  by  modeled  factors.  Or  stated  differently, 
it  tests  the  null  hypothesis  that  the  mean  of  a  sample  population,  the  residuals,  is  equal  to  a 
specified  value  /Tq  • 

The  t-statistic  is  found  by  taking  the  mean  of  the  sample  population  x,  minus  the  test 
value  jUo,  and  dividing  this  by  the  sample  standard  deviation  s/^n,  where  n  is  the  sample  size. 


A  two-stage  model  can  allow  implicit  correction  of  heteroskedastic  variance  in  a  dependent 
variable.  In  the  first  modeling  stage  we  control  for  additive  independent  variables  and  in  the 
second  stage  multiplicative  independent  variables.  This  approach  can  outperform  dependent 
variable  transformations,  as  it  incorporates  a  priori  knowledge  about  modeled  relationships. 
The  approach  yields  unbiased  results  as  long  as  the  independent  variables  in  the  first  stage  are 
uncorrelated  with  the  independent  variables  in  the  second  stage. 
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For  most  regression  models,  to  include  those  presented  here,  the  test  value  /Tq  is  zero,  with  the 
mean  of  the  errors  averaging  to  zero  by  construction.  The  t-statistic  is: 


x-fip 

s/yfn 


The  t-statistic  comes  from  a  Student's  t-distribution,  which  maps  directly  to  probability 
values  allowing  the  determination  of  statistical  significance.  Rearranging  the  t-test  equation, 
thresholds  can  be  defined  at  the  standard  levels  for  statistical  significance  0.10,  0.05,  and  0.01, 
which  corresponds  to  the  identification  of  a  change  from  the  baseline  with  90%,  95%,  and  99% 

certainty  16.  The  determination  of  the  thresholds  is  accomplished  by  first  finding  the  t- 
distribution  value  for  the  desired  confidence  level  p,  and  degrees  of  freedom  in  the  sample  size 
n  —  1.  The  t  value,  scaled  by  the  sample  standard  deviation  s/^^n  provides  upper  and  lower 
bounds  for  the  sample  mean  minus  the  test  value: 


t 


n-l,p 


<  X  p.Q  <  fn-l.p 


The  exact  likelihood  of  having  observed  a  change  can  be  found  by  solving  explicitly  for 
t;  all  the  required  sample  characteristics  are  known  (sample  mean  x,  number  of  events  in  the 
sample  n,  test  value  /Tq,  and  sample  variance  s/^n): 


X  =  Po  +  t 


n-l,p 


i^Analyst  selection  of  an  appropriate  significance  level  will  be  dictated  by  context.  It  must 
balance  type  1  error,  an  incorrect  rejection  of  the  null  hypothesis  and  type  11  error,  falsely 
rejecting  the  null  for  the  alternative  hypothesis  when  in  fact  the  null  hypothesis  is  true  (Pratt  et 
al,  2001). 
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Utilizing  the  detailed  approach,  Figure  7  shows  one  possible  visualization  of  significant 
unexpected  deviations  in  insurgent  activity  that  occurred  in  2012.  Significant  increases  are 
denoted  by  shades  of  red,  with  darker  reds  indicating  increased  statistical  confidence. 

Significant  decreases  are  denoted  by  shades  of  green,  with  darker  shades  indicating  increased 
statistical  confidence.  UB/LB  means  upper  and  lower  confidence  bounds,  gray  circles  represent 
non-significant  incidents.  While  the  underlying  analysis  leverages  daily  residuals.  Figure  7  plots 
moving  7-day  averages  of  enemy-initiated  attacks,  smoothing  the  trend  line  for  visual  analysis. 
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Figure  7,  Results  of  a  two-stage  least  squares  model  to  identify  unexpected  trends.^’ 


The  strength  of  this  proeess  is  that  in  reeurring  analysis  it  allows  us  to  readily  identify 
deviations  from  expected  insurgent  activity  as  they  occur.  With  the  identification  of  significant 
changes,  possible  explanations  for  these  shifts  are  as  follows: 

1)  An  atypically  severe  winter  might  have  caused  a  delay  in  the  start  of  the  fighting 
season. 

2)  A  poorer  harvest  likely  returned  insurgent  fighters  to  the  fight  earlier.  At  the  same 
time,  the  burning  of  Korans  at  Bagram  airbase  reaches  international  and  Afghan 
media  outlets  inciting  protests. 

3)  EIAs  decrease  significantly  surrounding  Eid  al-Eitr  as  insurgents  travel  to  and  from 
home  for  the  observance. 


Results  of  a  two-stage  least  squares  model  to  identify  significant  unexpected  trends  in 
nationwide  enemy-initiated  attacks  in  Afghanistan  in  2012.  Significant  increases/decreases  are 
denoted  by  shades  of  red/green,  with  darker  red/green  indicating  increased  statistical 
confidence.  UB/LB  means  upper  and  lower  confidence  bounds,  gray  circles  represent  non¬ 
significant  incidents 
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4)  Unseasonable  cold  temperatures  likely  led  to  lower  than  usual  expected  insurgent 
activities,  but  these  were  almost  immediately  followed  by  a  period  of  unseasonable 
warm  temperatures  that  probably  led  to  an  increase  in  activity  levels. 

Methods  for  Visualizing  Time  Series  Data 

Visualization  is  a  very  useful  tool  for  showing  changes  in  patterns  or  trends  that  might 
otherwise  go  unnoticed.  Standard  ways  to  display  changes  over  time  comprise  among  others  line 
charts,  bar  charts,  or  scatter  plots  such  as  shown  in  Figure  7  with  a  supplementary  color 
dimension  to  convey  additional  information,  in  this  case  statistical  significance.  Time  series  data 
can  be  categorized  by  considering  their  context  or  frame  of  reference  -  usually  distinguished  into 
abstract  and  spatial.  An  abstract  frame  of  reference  simply  refers  to  data  that  has  been  collected 
in  a  non-spatial  context,  i.e.,  without  connection  to  a  spatial  layout.  A  spatial  frame  of  reference, 
on  the  other  hand,  implies  the  existence  of  a  spatial  layout  for  the  data  set  -  it  allows  viewing 
space  as  some  kind  of  indexing  system  for  events  (Anselin,  1992;  Cressie,  1993;  Cressie  and 
Wilke,  2011). 

In  the  Afghanistan  Theatre  of  Operations  spatial  information  is  collected  -  its  geographic 
location  for  each  event.  Although  it  is  important  to  know  the  insurgent  activity  changes  over 
time,  it  is  equally  important  to  understand  where  insurgent  actions  occur,  their  spatial  structures, 
spatial  interactions,  and  the  population  that  is  impacted.  The  power  for  spatial  data  analysis  lies 
in  its  geo-relational  database  structure,  i.e.,  in  the  combination  of  value  information  and 
locational  information.  A  geographical  information  system  (GIS)  is  the  natural  tool  to  aid  in  the 
display  and  analysis  of  spatial  data.  The  advanced  display  capabilities  contained  in  a  GIS  can  be 
useful  for  the  visualization  of  processed,  geo-referenced  data,  or  the  results  of  statistical 
analyses.  These  spatial  visualizations  can  have  a  temporal  dimension.  In  Figure  8a  and  Figure 

8b  we  show  examples  for  two  static  geospatial  representations.  Figure  8a  displays  insurgent 
activity  in  Afghanistan's  central  Helmand  Province  relative  to  the  population  that  is  visualized  as 
a  background  layer.  The  location  of  each  individual  insurgent  attack  is  represented  by  a  colored 
point,  where  the  color  represents  one  of  the  two  time  periods  under  consideration  -  June  to 
September  2012  and,  for  comparability,  the  same  period  in  201 1 .  This  representation  shows  that 
insurgent  activity  moved  from  higher  population  densities  in  the  central  part  (during  the  2011 
time  period),  to  areas  with  lower  population  densities  in  2012.  Combined  team  operations  and  a 
high  operational  tempo  appear  to  have  had  a  considerable  impact  on  the  insurgent's  campaign  by 


Static  representations  visualize  the  data  in  still  images  (i.e.,  the  representations  do  no  changes 
automatically  over  time),  while  dynamic  representations  convey  the  time  dependency  of  the 
data  (i.e.,  representations  over  sequential  time  periods). 
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reducing  attacks  in  key  population  centers  in  Helmand  Province,  namely  in  Marjeh  District,  as 
well  as  Lashkar  Gah,  Nad'Ali,  and  western  Nahr-e  Saraj  Districts. 

Figure  8a  shows  another  static  representation  of  the  enemy  activity  for  central  Helmand 
Province.  Figure  8b  shows  the  difference  between  insurgent  attach  densities  for  2011  and  2012 
time  periods  for  this  same  region.  This  representation  reveals  a  relative  change  in  insurgent 
activity  from  one  year  to  another.  Representations  like  those  in  Figure  8a  and  Figure  8b  can 
greatly  improve  the  situational  and  contextual  understanding  of  both  the  analyst  and  more 
importantly  the  leadership  and  decision  makers. 


O  ElAsJun- Sep  2011 
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Figure  8a,  Static,  geospatial  scatter  plot  visualizing  individual  enemy-initiated  attacks. 
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Figure  8b,  Static  geospatial  density  plot  visualizing  changes  in  enemy-initiated  attacks 

More  robust  spatial  time  series  analysis  can  help  to  quantify  these  observed  trends  (Gaul, 
2013).  We  achieve  this  by  measuring  the  proximity  of  the  population  to  insurgent  activity,  or 
more  precisely  by  considering  the  average  number  of  Afghans  that  live  within  1  km  of  each 
enemy-initiated  attack.  A  geographical  information  system  is  utilized  to  conduct  the  laborious 
spatial  analysis. 

Figure  9  shows  the  result,  i.e.,  the  average  number  of  Afghans  in  proximity  of  an  attack 
by  month  for  all  of  Afghanistan  since  January  2010.  We  evidently  see  a  continuing  downward 
trend  in  this  metric  since  2010  when  the  average  number  of  Afghans  living  within  1  km  of  an 
enemy-initiated  attack  was  approximately  2,000,  while  in  November  2012,  around  1,300 
Afghans  were  objectively  affected  on  average.  Each  EIA  affects  (on  average)  fewer  Afghans,  or 
in  other  words,  the  insurgent  attacks  migrate  to  less  densely  populated  areas  where  they  are  close 
to  fewer  and  fewer  Afghans.  The  visualized  downwards  trend  is  statistically  significant. 
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Avg  Number  of  Afghans  Living  within  1km  of  each  Enemy-initiated  Attack 
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Figure  9,  Average  number  of  Afghans  living  within  1  km  of  each  enemy-initiated  attack 

(EIA). 

Modern  warfare  is  creating  vast  amounts  of  data.  Leveraging  this  information  across 
time  and  space  can  crucially  advance  operational  and  strategic  goals.  It  is  critical  for  analysts  to 
be  equipped  with  the  right  tools,  skills,  and  opportunities  to  process  and  manipulate 
spatiotemporal  data  generated  by  a  (military)  campaign,  thus  facilitating  expedient  analysis  of 
enemy  actions  and  blue  force  operations. 


Challenges 

During  our  time  at  ISAF  Headquarters  we  have  encountered  several  challenges  that  have 
impeded  our  ability  to  perform  effective  analysis.  They  include  the  cost  and  time  needed  to 
establish  an  operational  database,  i.e.,  data  collection,  access  to  the  data  and  data  sharing;  data 
quality  control  and  quality  assessment;  the  merging  of  data  sets)  and  access  to  analytic  software. 
Beyond  these  structural  challenges,  there  often  exists  a  gap  between  decision  makers  and 
analysts.  The  analysts  need  to  understand  the  issues  facing  the  top-level  leadership  in  order  to 
best  focus  their  analysis. 

In  addition  to  the  large  upfront  costs  of  establishing  a  database  structure  useable  across  a 
command,  there  is  the  time  and  manpower  needed  to  maintain  it,  e.g.,  collate  operational  reports 
into  the  database,  reconciling  differences  between  inputs,  manage  lost  or  incomplete  data,  etc. 
This  requires  planning  and  a  dedicated  funding  pool. 
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Across  a  theatre  of  operations  different  organizations  often  maintain  databases  or  data 
repositories  for  various  reasons,  such  as  the  collection  of  a  specific  type  of  information  that  is  not 
collected  elsewhere,  or  because  of  a  lack  of  knowledge  of  existing  databases.  This  can  cause  the 
analyst  additional  challenges,  the  primary  ones  being  1)  lower  standards  concerning  quality 
control  or  data  consistency  and  data  integrity  compared  to  the  authoritative  "command 
databases,"  and  2)  difficulties  in  assessing  and  accessing  the  data  .  The  ability  to  access  data  is 
crucial  to  analysis  and  assessment  -  additional  non-authoritative  databases/repositories,  although 
potentially  well  intended,  can  have  counterproductive  effects. 

Understanding  data  quality  and  the  constraints  it  places  on  subsequent  analysis  is  an 
important  prerequisite  to  formulating  an  assessment  and  making  recommendations.  The  ability 
to  control  data  quality  by  cleaning,  processing,  and  organizing  data  is  a  necessary  skill  for 
analysts.  In  some  cases  analysis  that  could  be  done  should  be  avoided  because  underlying  data 
inconsistencies  would  lead  to  conclusions  too  unreliable  to  guide  decision  making. 

The  data  necessary  for  an  analysis  may  not  exist  in  one  repository.  The  ability  to  join 
large  datasets  is  what  enables  many  analyses.  Combining  datasets  and  looking  across  previously 
different  lines  of  effort  allows  for  the  identification  of  formerly  undetected  interdependencies 
and  interrelationships  that  can  bring  important  campaign  insights. 

Access  to  analytic  software  is  the  final  major  constraint  for  the  analyst.  The  tools  to 
manage  and  manipulate  large  quantities  of  spatiotemporal  data  can  be  expensive.  Funding  such 
tools  requires  improved  awareness  of  their  operational  benefits.  Similarly,  system  protocols  and 
security  related  constraints  often  hinder  access  to  required  software  such  as  Microsoft  Excel's 
VBA  (Visual  Basic  for  Applications)  or  tools  publically  available  for  statistical  analysis  such  as 
R  data  analysis  software. 

Operational  analysis  can  provide  commanders  critical  information  about  the  campaign. 

In  this  line,  analysts  need  to  link  directly  into  the  pulse  of  the  campaign  and  understand  current 
commander  priorities,  while  leadership  should  seek  to  remove  hurdles  hindering  analysis.  A 
good  analyst  can  often  help  transform  commander's  priorities  and  guide  valuable  research  that 
informs  future  decisions. 

Summary  and  Conclusion 

Leveraging  the  vast  amounts  of  data  collected  across  time  and  space  in  modem  military 
campaigns  can  cmcially  advance  operational  and  strategic  goals.  It  is  critical  for  analysts  to  be 
equipped  with  the  right  tools,  skills,  and  opportunities  to  process  and  manipulate  these  data. 

Time  series  analysis  is  one  of  these  highly  valuable  tools  and  can  generate  crucial  insights  for  a 
command,  enabling  the  assessment  of  operational  effectiveness  and  developments  in  insurgent 
tactics.  By  combining  time  and  spatial  domains  it  is  possible  to  identify  further  relationships  and 
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opportunities  that  can  be  exploited.  In  this  chapter  we  have  presented  the  fundamentals  of  time 
series  analysis  and  discussed  some  of  its  uses  stemming  from  our  experiences  of  working  in  the 
AAG  at  ISAF  Headquarters  in  Kabul,  Afghanistan.  We  concluded  by  presenting  some  of  the 
assessment  challenges  encountered  during  the  ISAF  campaign;  lessons  that  may  help  future 
analysts  and  commanders  better  structure  their  campaigns  for  success. 
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3.8  Summary 


There  are  numerous  analytieal  teehniques  one  ean  use  to  exeeute  analysis.  One  does  not  have  to 
be  an  expert  in  every  teehnique,  but  one  should  know  whieh  teehnique  is  appropriate  for  a  given 
problem.  If  a  deployed  ORSA  analyst  finds  that  a  problem  requires  a  speeifie  teehnique  where 
time,  skill,  or  software  availability  is  at  issue,  he  or  she  ean  request  analytieal  support  from  a 
reaehbaek  ageney  sueh  as  CAA  or  tap  into  his  or  her  personal  ORSA  network. 
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CHAPTER  4  COMMUNICATING  CLEARLY 

4.1  Purpose 


The  purpose  of  this  ehapter  is  to  provide  ORSA  analysts  with  best  praetiees  for  representing  data 
and  assoeiated  analyses  in  a  elear,  aeeurate,  and  understandable  way. 

4.2  Constraints,  Limitations,  and  Assumptions  (CLA) 


CLA  bound  (seope)  a  study  effort  by  identifying  what  must  (or  must  not)  and  ean  (or  eannot)  be 
aeeomplished;  they  frame  the  study  spaee  and  set  the  stage  for  the  study  team's  methodology 
development;  they  serve  as  a  "eontraet"  between  the  study  sponsor  and  the  study  team;  and  they 
provide  a  basis  for  the  sponsor  to  reeoneile  the  study  results.  CLA  provide  the  framework  for 
both  the  study  team  and  the  study  sponsor  to  understand  the  eonditions  under  whieh  a  study's 
results  are  applieable. 

4.2.1  Constraints 

Constraints  are  restrictions  imposed  by  the  study  sponsor  that  limit  the  study  team's  options  in 
conducting  the  analysis  and/or  study.  The  study  tasker  and  sponsor's  guidance  may  place  bounds 
on  the  study  by  specifying  the 

•  date  by  which  the  study  must  be  completed; 

•  organizations  to  participate  in  the  study; 

•  force  structure(s)  and  force  year(s)  to  consider; 

•  type(s)  of  combat  operations  to  consider;  and 

•  scenario(s),  threats,  and  environments  to  consider. 

These  bounds  may  limit  one's  ability  to  conduct  a  full  investigation  and  analysis  of  the  issues  at 
hand.  One  needs  to  discuss  changes  to  any  constraint  with  the  study  sponsor  prior  to  removing 
or  adjusting  the  constraint. 

4.2.2  Limitations 

Limitations  are  restrictions  that  limit  one's  ability  to  meet  the  study  objectives  or  fully 
investigate/analyze  the  study  issues  within  the  bounds  of  the  study's  sponsor.  Limitations  may 
consist  of  the  following 

•  concept  immaturity; 

•  access  to  study  information,  e.g.,  information  that  requires  special  clearances; 

•  availability  of  data; 

•  number  and  type  of  available  scenarios;  and, 

•  available  models  and  their  capabilities. 

One  should  not  label  something  as  a  limitation  that  one  imposes  on  oneself.  If  one  can  overcome 
a  limitation  by  having  the  study  sponsor  change  a  constraint,  one  should  recommend  that  change 
to  the  sponsor  and  identify  how  it  benefits  the  analysis/study. 
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4.2.3  Assumptions 

Assumptions  are  statements  related  to  the  analysis/study  that  are  taken  as  true  in  the  absenee  of 
facts,  often  to  accommodate  limitations. 

Clearly  define  assumptions  used  in  one’s  analysis.  Assumptions  are  an  integral  part  of  any 
analysis.  One  needs  to  clearly  define  and  state  all  assumptions  associated  with  the  analysis. 
Higher  headquarters  or  written  plans  provide  the  assumptions  associated  with  the  analysis.  One 
usually  identifies  these  assumptions  up-front  to  help  shape  the  method  of  analysis.  One  may  also 
discover  other  needed  assumptions  as  the  analysis  progresses.  When  one  discovers  additional 
assumptions  during  the  analysis,  one  needs  to  discuss  these  assumptions  with  the  sponsor  to 
determine  their  relevance  or  to  obtain  additional  data  that  would  eliminate  them.  Whether  stated 
up-front  or  determined  over  the  course  of  analysis,  one  needs  to  make  all  recipients  of  the  final 
analytical  product  aware  of  all  assumptions  by  incorporating  them  into  the  final  presentation  or 
report. 

State  how  the  assumptions  will  affect  analysis  results.  It  is  important  to  state  how  each 
assumption  will  affect  one's  analysis  and  study  results.  One  should  only  include  those 
assumptions  that  have  an  impact  on  the  analysis  and  results.  If  one  cannot  explain  the  effect — or 
there  is  no  effect  on  the  results — one  should  not  include  the  assumption.  One  should  explain  all 
assumptions  during  one's  presentation  or  in  the  notes  section  of  the  final  product  or  report.  The 
following  are  examples  of  relevant  assumptions  made  in  past  analyses  conducted  in  theater: 

•  Specific  sensor  system  had  line  of  sight  to  view  significant  activities. 

•  Attrition  rate  of  6.4%. 

•  International  funds  would  continue  at  the  same  level  for  the  next  x  number  of  years. 

Based  on  the  available  data  at  the  time  of  the  analysis,  one  makes  assumptions  to  allow  analysis 
to  continue.  If  additional  data  become  available  during  the  analysis,  one  should  include  them  to 
provide  greater  fidelity  and  insight.  Assumptions,  such  as  those  listed  above,  assist  in  the 
development  of  a  data  collection  plan  and  enhance  future  iterations  of  the  analysis. 

4.3  Roles  and  Tenets  of  CL  A 


CLA  are  critical  to  a  successful  study.  They 

•  bound  (scope)  a  study  effort  by  identifying  what  must  (or  must  not)  or  can  (or  cannot)  be 
accomplished; 

•  frame  the  study  space  and  set  the  stage  for  the  study  team's  method-development; 

•  serve  as  a  contract  between  the  study  sponsor  and  the  study  team;  and 

•  provide  a  basis  for  the  sponsor  to  reconcile  the  study  results. 

CLA  must  be 

•  acceptable,  i.e.,  generally  agreed  upon  by  all  study  participants; 

•  evolving,  i.e.,  continually  reviewed  and  adjusted  as  the  study  effort  matures; 

•  necessary,  i.e.,  enable  the  study  effort;  and 
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•  valid,  i.e.,  sound  and  supportable. 

4.3.1  State  All  Caveats 

One  must  clearly  state  all  caveats  associated  with  the  data  and  analysis.  Caveats  are  warnings  or 
cautions  that  may  include  one's  interpretation  of  data,  interpretation  of  terminology,  and  one's 
graphical  representation  of  results.  One  may  address  caveats  in  the  form  of  limitations.  One 
should  address  caveats  up-front  to  give  one's  audience  a  greater  perspective  while  they  are 
reviewing  the  analysis  and  results.  For  example,  one  should  include  a  caveat  associated  with  a 
specific  graphical  representation  of  results  as  a  text  box  or  bullet  on  the  same  slide  as  the  graphic 
to  avoid  misrepresentation  and/or  misunderstanding  of  the  results. 

The  following  are  examples  of  caveats  from  OIF  and  OFF: 

•  Population  data  were  extrapolated  from  the  census  conducted  in  1975. 

•  All  levels  of  armoring  (1,  2,  &  3)  were  included  in  the  up-armored  high  mobility 
multipurpose  wheeled  vehicle  (HMMWV)  category. 

•  Only  coalition  forces  (CF)  casualty  data  were  included. 

•  Responses  for  Afghan  National  Development  Program  (ANDP)  opinion  survey  4.0  were 
worded  differently  from  previous  surveys.  Responses  were  binned  to  compare  with 
ANDP  surveys  1.0  -  3.0. 

Caveats  such  as  these  inform  the  sponsor  to  begin  collection  on  specific  data  requirements  to 
enhance  future  iterations  of  ANDP  analysis. 

4.3.2  Presenting  Results 

Know  the  audience.  Before  one  presents  a  brief  on  analytical  results,  he  or  she  needs  to  know 
his  or  her  audience.  Knowing  one's  audience  assists  one  in  knowing  what  level  of  discussion  to 
have  on  the  analytical  techniques  used  in  the  analysis. 

Avoid  becoming  too  technical.  If  one  wants  senior  leaders  to  accept  one's  technical  study 
results  at  the  highest  levels,  one  must  communicate  insights  as  simple  statements,  without 
becoming  too  technical.  One  should  not  assume  that  the  decision  maker  will  sift  through 
briefing  data  to  find  the  answers.  The  purpose  of  a  briefing  is  not  to  show  the  decision  maker  all 
the  data  and  expect  him  or  her  to  find  the  answer.  One  must  realize  that  most  decision  makers 
are  not  technical  experts.  If  one  displays  too  much  technical  data,  one  further  complicates  an 
already  complex  issue.  One  must  package  one's  results  to  provide  insight  and  recommendations 
to  decision  makers.  Decision  makers  focus  on  decision  making,  not  technical  fascination. 

Present  bottom  line  up  front.  One  should  clearly  state  the  purpose  of  one's  brief/presentation  at 
the  beginning  in  the  form  of  a  purpose  statement  or  problem  statement.  One  should  make  these 
statements  concise  and  develop  them  jointly  with  the  study  sponsor/study  director. 

Use  a  logical  presentation  methodology.  One's  presentation  should  have  a  logical  flow  to 
allow  one's  audience  to  understand  the  purpose,  problem,  analysis,  and  results.  One  should 
begin  one's  presentations  with  the  purpose  and/or  problem  statement,  i.e.,  "bottom  line  up  front." 
This  technique  enables  one's  audience  to  understand  the  end  state  and  results  while  the 
discussion  is  unfolding.  Any  graphical  representation  of  data  or  analysis  should  include  a  "take 
away"  bullet.  One  should  include  a  bullet  comment  on  the  most  important  reason  for  displaying 
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the  data/results  in  a  given  manner.  This  bullet  should  stand  out  using  a  color  scheme,  larger  bold 
text,  or  some  type  of  symbol  to  draw  the  audience's  attention  to  this  important  fact. 

If  there  is  specific  information  that  should  remain  displayed  throughout  the  brief,  e.g.,  a  map  of 
the  operational  area  or  a  legend  of  a  color  scheme),  one  should  create  separate  visual  displays  to 
make  this  possible. 

The  delivery  of  one's  brief  is  just  as  important  as  the  brief  itself.  One  should  rehearse  one's  brief 
to  increase  one's  confidence,  correct  one's  mistakes,  and  make  changes  to  the  flow  of  one's 
presentation.  One  should  take  the  time  to  ensure  all  technology  is  operating  properly  (including 
replacing  the  batteries  in  the  remote,  the  bulbs  in  the  projector,  the  length  of  extension  cords  for 
the  computer  equipment,  the  volume  on  the  sound,  etc).  If  a  remote  is  not  available,  one  should 
arrange  for  someone  else  to  work  the  slide  show  presentation  (if  one  has  time  to  rehearse,  this  is 
always  a  good  idea  because  it  limits  one's  distractions  and  keeps  one's  audience  focused  on  the 
presentation).  One  can  have  the  best  analytical  product  in  the  world,  but  if  one's  delivery  is  poor, 
the  decision  maker  may  discount  one's  efforts. 

After  giving  the  presentation,  one  should  re-assess  the  overall  briefing  and  determine  if  one 
achieved  one's  stated  goals.  One  should  determine  if  follow-on  presentations  or  meetings  are 
necessary  and  appropriate. 

Prepare  slides  to  stand  alone.  One's  slides  should  contain  backup  notes  and  potentially  stand 
alone.  Commanders  and  staffs  at  all  levels  may  extract  specific  slides  from  one's  presentation 
for  other  presentations/meetings/working  groups,  etc.  Providing  backup  notes  with  one's  slides 
limits  misuse  and  misinterpretation. 

Include  the  appropriate  classification  with  each  hullet.  When  incorporating  classified 
information  into  one's  presentation,  one  should  use  a  Security  Classification  Guide  or  properly 
marked  source  document(s)  to  determine  the  classification  of  information  incorporated  into  their 
presentation.  Every  portion  in  a  classified  presentation  should  be  marked  for  classification,  and 
every  slide  should  contain  a  classification  banner  indicating  the  highest  level  of  classification 
found  on  that  slide.  Reference  the  governing  regulations  or  memorandums  on  marking  of 
classified  information  that  pertain  to  your  working  location. 

Use  the  hest  method  of  presentation.  Consider  one's  audience  in  determining  the  best  method 
to  display  one's  results.  One  should  remember  one  is  conducting  the  analysis  for  the  customer. 

In  some  cases,  this  method  is  command-driven.  If  it  is  not,  the  following  are  helpful  hints  on 
best  practices  used  in  theater. 

Present  information  in  the  clearest  manner.  One  should  not  make  slides  so  complex  that  the 
audience  has  difficulty  understanding  the  main  point  of  one's  slides. 

Use  a  comhination  of  graphs  or  tables  with  accompanying  hullets.  One  should  highlight  the 
"take  away"  of  the  slide.  One  should  use  colored  text  boxes  and  larger  fonts  to  make  important 
points  stand  out  from  other  information  on  the  slide.  When  one  uses  a  common  color  scheme 
throughout  the  presentation,  one's  audience  quickly  recognizes  the  main  point  of  each  slide. 

Properly  lahel  graphical  representation  of  results.  One  should  label  all  aspects  of  each  graph, 
i.e.,  x-axis,  y-axis,  title,  and  legend.  One  should  clearly  define  data  table  columns  and  rows. 

One  should  include  the  data  source  with  all  graphs  and  tables  to  enable  one's  audience  to  find  the 
original  data  for  follow-on  analysis  and/or  validation. 
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Simplify  mathematical  concepts.  One  should  know  one's  audience  when  explaining  the 
mathematical  concepts  behind  the  analysis.  As  a  rule,  one  should  present  concepts  simply  and 
have  the  mathematical  computations  available  in  one's  back-up  slides  to  present  to  senior  leaders 
who  request  additional  information. 

Graphically  display  results  and  conclusions.  Graphical  displays  are  often  more  powerful  than 
the  technical  analysis  behind  them.  One  should  take  extra  time  to  display  one's  information  in 
such  a  way  that  one's  audience  easily  grasps  the  main  points.  When  one  incorporates  tables, 
charts,  or  graphs  into  a  PowerPoint  presentation,  one  should  paste  them  as  pictures.  One  does 
this  by  using  the  Edit,  Paste  Special  command  in  PowerPoint.  This  prevents  anyone  else  from 
manipulating  one's  data  and  charts.  It  also  reduces  the  size  of  one's  presentation  and  makes  it 
easier  to  send  electronically.  The  following  are  some  helpful  hints  for  displaying  results: 

•  Visuals  are  more  powerful  in  color.  One  should  use  a  specific  color  for  the  critical  points 
or  "take-a-ways"  that  require  audience  attention.  One  should  limit  the  number  of  colors 
per  graphic  and  keep  colors  consistent  across  one's  entire  presentation.  Each  graphic 
must  contain  a  detailed  legend  that  defines  one's  color-coding  methodology.  Ensure  your 
primary  audience  member,  e.g.,  the  commander,  is  not  colorblind,  which  could  render 
your  message  ineffective. 

•  One  should  avoid  confusing  one's  audience;  one  should  keep  scales  constant  across 
multiple  charts  on  the  same  topic. 

•  After  completing  the  analysis,  one  should  determine  the  type  of  graphic  that  best 
communicates  the  results  and  main  "take-a-way"  points,  e.g.,  text  charts,  tables,  line 
charts,  dual-axis  charts,  surface  charts,  vertical  bar  or  column  charts,  and  pie  charts). 

•  Text  charts  are  the  easiest  type  of  graphic  to  prepare.  However,  one  should  remember 
that  words  or  bullets  are  not  visuals.  Pictures,  clouds,  and  arrows  used  in  conjunction 
with  words  create  visual  effects  that  communicate  the  information  or  results  better.  Text 
charts  are  effective  for  introducing  topics,  such  as  the  methods  one  used  in  the  analysis 
and  important  ideas  or  results.  One  should  keep  one's  text  charts  as  concise  as  possible; 
one  should  use  bullet  statements  rather  than  entire  sentences  and  paragraphs. 
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Figure  10.  ANDP  survey  model 

Figure  10  is  a  text  ehart  displaying  the  survey  model  of  the  ANDP.  One  can  use  tables  to  display 
raw  data  or  study  results;  however,  there  are  better  methods  for  displaying  comparisons  or 
trends.  One  can  use  a  table  in  conjunction  with  other  graphics  if  the  exact  numbers  are  relevant 
to  the  "take-a-way"  one  wants  to  achieve  on  a  particular  slide.  When  one  uses  tables,  one  should 
clearly  label  all  columns  and  rows  (see  Figure  11). 


Tables  Show  Specific  Data 

Annual  Spending  ($6) 
Fiscal  Year 


FY 

1965 

1970 

1975 

1980 

1985 

Procurement  and  RDTE 

7.6 

9.0 

10.8 

13.4 

19.0 

Personal  Costs 

9.1 

11.3 

13.8 

19.5 

26.5 

Other 

4.9 

5.8 

6.7 

8.2 

10.8 

Figure  11.  Annual  ANDP  spending  per  fiscal  year 
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Date 


Figure  12.  Numbers  of  attacks  and  casualties 

Line  or  curve  charts  effectively  show  trends  and  time  series  data.  Line  charts  display  multiple 
lines  on  a  single  graph.  One  should  clearly  distinguish  each  line  with  differing  colors  or  patterns 
(solid,  dashed,  etc).  One  should  include  a  legend  that  defines  colors  and  patterns. 

Figure  12  is  a  sample  line  chart  of  notional  attack  and  casualty  data.  The  chart  defines  the  x-  and 
y-axes,  and  includes  a  legend  to  specify  the  representation  of  each  line.  The  title  specifies  the 
exact  category  of  the  data,  e.g.,  attacks  and  casualties. 
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Attacks  ^"Casualties  ^"Casualties  per  Attack 


Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec 


Month 


Figure  13.  Casualties  per  attack 

Dual  axis  charts  show  different  data  with  a  varying  degree  of  magnitude  between  the  data  sets. 
In  Figure  13,  the  ORSA  analyst  uses  a  red  line  to  display  notional  easualties  per  attaek.  The 
dual-axis  chart  presents  all  pertinent  information  in  a  clear  and  understandable  way. 
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Figure  14.  Numbers  of  participating  ISF/US/CF  security  forces 

In  Figure  14,  the  ORSA  analyst  uses  a  surfaee  ehart  to  display  data  for  the  Iraqi  Seeurity  Forees 
(ISF)  Update  eondueted  by  Multi-National  Forces-Iraq  (MNF-I).  A  surfaee  chart  is  a  single  line 
chart  with  the  area  shaded  under  the  line.  The  shading  emphasizes  the  size  of  the  total  amount 
rather  than  the  differences  or  changes  in  the  amounts. 
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Figure  15.  The  answer  "Road  improvements  would  most  improve  our  lives?" 

Figure  15  is  a  column  chart  used  to  display  the  results  of  the  answer  "road  improvements"  to  the 
question  in  the  ANDP  opinion  surveys  2.0  and  3.0 — "What  is  the  one  thing  that  would  most 
improve  your  life?" 

A  vertical  bar  or  column  chart  displays  values  and  series  groups  as  sets  of  vertical  columns 
grouped  by  category.  The  height  of  each  column  represents  each  respective  quantity  and  its 
variation  across  different  categories,  e.g.,  time  intervals  or  geographic  regions.  Excel  offers 
different  variations  of  this  type  of  chart,  to  include  the  stack  column  chart,  the  percentage 
column  chart,  and  the  option  to  graph  multiple  categories  across  the  same  intervals  of  time  or 
geographic  regions  by  displaying  each  as  a  different-colored  column.  Each  graph  should  include 
a  legend  that  defines  the  visual  coding. 
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Figure  16.  The  age-percentages  of  ANDP  3.0  survey  respondents 

Figure  16  displays  age  eategory  pereentages  for  ANDP  3.0.  A  pie  ehart  is  used  to  show  a 
eomposite  whole  and  its  proportions.  Pie  sliees  are  eompared  with  each  other  and  with  the 
whole.  Each  slice  is  represented  with  color  and  a  legend  is  used  to  explain  the  color-coding 
methodology. 


DAHB 


COMMUNICATING  CLEARLY  •  59 


CAA-20 15094 


Goal  Programming  Results  (Notional  Distribution) 
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Figure  17,  MEDEVAC  helicopter  response  times  and  number  of  events 

Figure  17  shows  notional  Afghanistan  MEDEVAC  data  and  helicopter  response  times  created 
with  ArcGlS.  Shading  represents  the  various  response  times  and  red  dots  represent  MEDEVAC 
events. 
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Figure  18  is  an  example  of  a  quad  chart  that  summarizes  and  highlights  trends  on  a  specific 
question  from  the  ANDP.  Across  all  four  iterations  of  the  ANDP  survey,  pollsters  asked,  "How 
is  the  security  situation  in  your  village/city?"  This  information  fed  indicators  and  measures  of 
effectiveness  (MOEs)  to  determine  if  U.S./host  nation  (HN)/CF  were  meeting  campaign  plan 
objectives. 


As  a  deployed  ORSA  analyst,  one  most  commonly  uses  the  "Quad"  chart  to  summarize  project 
status.  The  Quad  chart  divides  a  slide  into  four  sections  that  usually  consist  of  (1)  the 
purpose/problem  statement,  (2)  methods,  (3)  timeline,  and  (4)  results/insights.  One  can  adjust 
these  sections  to  fit  one's  specific  projects  or  quickly  summarize  one's  current  work  effort.  One 
can  share  these  charts  with  agencies  and  other  analysts  who  are  interested  in  this  work  or 
collaborating  on  a  reachback  project.  One  can  use  Quad  charts  to  show  relationships.  For 
example,  in  the  assessment  process,  one  may  find  this  type  of  chart  helpful  in  highlighting 
relationships  between  different  indicators  and  MOEs,  or  increases/decreases  in  a  specific  MOE. 
One  can  amend  the  "Quad"  chart,  e.g.,  "Tri"  or  "Bi"  chart)  as  the  situation  requires. 

4.4  Summary _ 


This  chapter  presented  best  practices  for  representing  data  and  associated  analyses  in  a  clear, 
accurate,  and  understandable  way.  It  presented  the  differences  between  constraints,  limitations, 
and  assumptions,  and  the  importance  of  each.  And  finally,  it  highlighted  the  importance  of  using 
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geospatial  analysis  as  a  powerful  tool  to  complement  other  visual  tools  for  displaying  one's 
analysis. 
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CHAPTER  5  PUBLIC  PERCEPTION  SURVEY  DEVELOPMENT 
AND  ANALYSIS 

5.1  Introduction 


This  chapter  will  discuss  the  value  of  publie  perception  surveys,  walk  through  the  process  to 
design  and  execute  a  survey,  review  analysis  and  presentation  methods,  and  eonelude  with  a 
discussion  on  the  limitation  of  survey  data  and  additional  resources. 

In  operational  environments  where  the  indigenous  population  is  the  center  of  gravity,  deeision 
makers  consider  the  population's  perspective  in  their  planning  calculus.  Commanders 
continually  make  decisions  to  address,  persuade,  and  change  public  perceptions  of  key  aspects 
and  focus  areas  of  the  nation-building  process,  sueh  as  reconstruetion,  governance,  security,  etc. 
Surveys  play  an  important  role  in  understanding  the  population's  pereeption,  a  unique  data  point 
to  understanding  the  overall  operational  environment. 
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Figure  19,  Survey  process  map 

A  broad  overview  of  the  entire  proeess  for  designing,  executing,  and  presenting  public 
perception  data  is  shown  here. 

5.2  Practical  Apiilications  and  Examples 

Public  perception  data  ean  inform  a  broad  range  of  deeisions,  spanning  from  the  strategic  level  to 
more  tactical  decisions.  This  section  lays  out  examples  of  the  different  ways  in  which  survey 
data  can  be  used. 

Assessment  Indicators 

Commanders  use  assessments  to  determine  whether  progress  has  been  made  towards  achieving 
stated  objectives.  Survey  data  are  often  used  as  a  data  source  for  indieators  as  part  of  the 
assessment  framework.  Data  collected  over  time  can  show  how  a  population's  sentiment  has 
changed  over  time,  whether  towards  extremist  groups,  the  U.S.,  a  U.S-Coalition  military 
presence,  or  the  host  nation  government. 

Example  1.  International  Secnrity  Assistance  Force. 

ISAF’s  Afghan  National  Quarterly  Assessment  Report  (ANQAR),  a 
nationally  representative  public  perception  survey,  was  used  as  one 
indicator  to  inform  assessments  of  the  ISAF  mission  by  showing  the 
trends  of  Afahans’  nercention  of  seeuritv  over  time. 
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Inform  Planning  Requirements 

Survey  data  are  also  used  to  inform  the  development  and  adjustment  of  operations,  particularly 
those  related  to  civil  affairs  and  military  information  support  operations  (MISO).  Operators  can 
identify  target  populations  for  potential  operations,  identifying  areas  of  opportunity  or  need 
among  the  population.  This  includes  identifying  pockets  of  the  population  that  are  sympathetic 
towards  violent  extremists  and  understand  why  they  are  sympathetic.  Then,  using  demographic 
and  media  usage  information  gleaned  from  the  survey  data,  analysts  can  design  a  specific, 
targeted  MISO  series. 


Example  2,  Special  Operations  Command  Africa 

CAA  used  Department  of  State’s  nation-wide  surveys  in  Niger  to  inform 
the  planning  decisions  of  Special  Operations  Command  Africa,  who  was 
looking  to  build  an  airbase  in  the  Agadez  Region.  The  survey  data 
provided  insights  on  the  public’s  sentiment  towards  the  United  States,  as 
well  as  identified  areas  of  opportunity,  such  as  infrastructure 
improvement  for  the  United  States  to  build  up  good  will. 


Example  3,  Joint  Special  Operations  Task  Force  -  Philippines 

CAA’s  Southern  Philippines  Public  Perception  Survey  identified  pockets 
of  the  Mindanao  population  where  the  people  were  unfamiliar  with  the 
Bangsamoro  Peace  Process.  This  information  was  used  to  develop  a 
MISO  series  to  education  the  population  where  it  was  needed,  conducted 
in  coordination  with  Philippine  counterparts. 


Before  Starting 

Know  the  ultimate  objective  of  the  survey  -  what  decision(s)  is  it  informing?  What  are  the  data 
points  stakeholders  want  from  the  survey?  What  information  will  it  provide  that  other  data 
sources  cannot?  Know  who  the  primary  stakeholders  are,  and  what  information  they  want  from 
the  survey.  Ensure  those  expectations  are  realistic  and  fall  within  the  scope  of  a  public 
perception  survey. 

Process  of  Designing  and  Executing  a  Survey 

The  involvement  of  the  ORSA  analyst  in  the  survey  process  will  depend  on  the  role  of  the 
supported  command;  it  may  range  from  designing,  implementing,  managing  the  entire  survey 
process  to  simply  analyzing  data  collected  through  an  already-existing  survey  channel.  This 
sections  walks  through  that  entire  process. 

If  operating  under  a  tight  timeframe  or  with  limited  resources,  there  may  not  time  to  complete  all 
these  steps  (pre-tests,  back  translations  of  the  questionnaire,  etc).  Local  contractors  will  almost 
always  do  the  actual  fieldwork;  locals  conducting  surveys  to  local  people  in  their  own  local 
dialect  are  much  better  received  than  if  U.S.  personnel  were  to  conduct  the  surveys. 

A  note  on  working  with  contractors:  Be  certified  as  a  Contracting  Officer's 
Representative  (COR).  This  provides  the  ability  to  communicate  directly  with  the 
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contractors,  instead  of  going  through  the  Contracting  Officer  (KO).  The  KO  will  almost 
certainly  not  be  a  subject  matter  expert  (SME)  in  publie  perception  surveys,  so  part  of  the 
COR  role  will  be  to  work  with  the  eontraetor  in  the  development  of  the  sampling  plan 
and  questionnaire,  cheek  in  with  them  to  ensure  they  are  following  the  aecepted  proeesses 
and  protocols,  and  follow  up  directly  with  the  eontraetor  if  problems  with  the  data  are 
identified,  e.g.,  incorrect  coding,  responses  not  translated  to  English,  etc.,.  Additionally, 
the  COR  will  review  the  Statement  of  Work  before  it  is  finalized  to  be  sure  the  details 
comport  with  the  requirements.  This  ineludes  sample  size,  questionnaire  length, 
sampling  plan,  training  of  interviewers,  format  of  data  delivery,  timeline  for  exeeution, 
data  coding  requirements,  and  any  required  final  products  from  the  eontraetor. 

Sampling  Plan  and  Methodology  Development 

Professional  polling  companies  have  the  expertise  to  reeommend  the  best  polling  methodology 
based  on  the  survey's  purpose.  If  one  uses  a  eontraeting  ageney,  inspeet  the  sampling  plan  prior 
to  fieldwork  to  ensure  it  comports  to  acceptable  standards.  Considerations  include: 

•  Demographics.  Typically,  the  survey  sample  should  be  representative  of  the  population, 
as  dietated  by  an  authoritative  souree,  such  as  a  census.  Depending  on  the  purpose  of  the 
survey,  the  sample  may  need  to  focus  on  a  particular  portion  of  the  population.  For 
example,  a  survey  looking  at  the  influence  of  violent  extremist  organizations'  messaging 
may  focus  on  specific  vulnerable  populations  as  determined  by  the  survey  stakeholders. 

•  Random  Sampling.  In  order  to  limit  the  bias  assoeiated  with  surveys,  random  sampling 
needs  to  oecur  at  every  level  of  the  survey  proeess,  from  the  selection  of  towns  and  cities 
to  the  actual  people  one  will  interview.  Standard  seleetion  methods  usually  suffiee; 
however,  there  may  be  times  when  one  needs  to  eonsider  "non-standard"  seleetion 
methods. 

•  Sample  Size.  Even  when  a  survey  controls  for  demographics  and  random  sampling,  there 
is  a  margin  of  error  associated  with  any  opinion  poll.  There  are  many  methods  for 
calculating  this  margin  of  error  (eaeh  with  underlying  assumptions),  the  most  common 
involves  sample  size  and  confidenee  level.  The  equation  for  the  margin  of  error 
caleulated  at  the  95%  confidence  level  is: 

1  96 

%  Error  =  ,  where  n  represents  the  sample  size. 

Much  analysis  is  done  with  subsets  of  the  sample  -  male/female,  geographic,  and  religious 
breakdowns,  etc.  Keep  in  mind  that  each  of  these  sub-groups  will  have  larger  margins  of  error  as 
the  sample  size  being  analyzed  shrinks.  Make  sure  to  account  for  statistical  significance  in  the 
final  analysis  and  presentation  -  do  not  highlight  differences  that  are  not  meaningful. 

Questionnaire  Design 

Designing  a  questionnaire  should  be  a  team  effort,  with  involvement  from  all  stakeholders  to 
ensure  their  subject  areas  are  covered,  be  it  security,  governance,  development,  ete.  Typically, 
sections  on  demographics  as  well  as  media  usage  are  standard.  Whenever  possible,  one  should 
take  advantage  of  the  contracted  agency's  expertise,  both  in  polling  as  well  as  regional  expertise, 
to  assist  with  question  wording  and  understand  relevant  cultural  considerations. 

When  designing  a  questionnaire,  there  are  several  things  to  keep  in  mind: 
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•  Length.  Cost  will  limit  the  number  of  questions;  consider  interviewer  and  respondent 
fatigue  when  adding  questions.  Do  not  be  unnecessarily  redundant. 

•  Order.  Ask  about  awareness  of  an  issue  before  asking  an  opinion  of  that  issue.  Group 
questions  thematically. 

•  Continuity.  Keep  as  many  questions  from  previous  waves  as  possible  to  enable  trend 
analysis. 

•  Language.  Keep  it  simple.  Plain  language,  clear,  and  direct.  Consider  what  nuances 
may  be  lost  in  translation. 


_ Example  4.  Question  Design _ 

Bad  Question;  Do  you  think  the  government  is  doing  a  good  job  at  fighting  corruption  and 
providing  services  for  its  people? 

This  question  combines  multiple  sentiments  (fighting  corruption  and  providing 
services)  into  one,  which  limits  the  respondent.  The  respondent  may  think  the 
government  is  good  at  one  thing  and  not  the  other  -  the  data  resulting  from  this 
question  would  not  show  this.  Additionally,  words  should  be  specific  enough  that 
every  respondent  interprets  the  question  the  same  way.  What  is  meant  by  services? 
It  could  mean  a  broad  range  of  things,  and  this  question  would  not  give  meaningful 
data  to  act  upon. 

Better  Question:  Please  tell  me  whether  the  government  is  doing  a  very  good,  somewhat 
good,  neither  good  nor  poor,  somewhat  poor,  or  very  poor  job  for  each  of  the  following 
items; 

i.  Fighting  corruption. 

ii.  Providing  security  services, 
hi.  Providing  education  services, 
iv.  Providing  healthcare  services. 

V.  Providing  transportation  services, 
vi.  Providing  electricity  services. 

This  question  asks  about  different  tasks  of  the  government,  separately  and 
specifically.  The  four  answer  choices  offer  more  granularity  for  respondents. 


There  are  several  types  of  questions.  These  are  some  general  categories: 

•  Closed-Ended  Questions:  Respondents  are  limited  to  the  answer  choices  provided  by  the 
interviewer.  The  answer  choices  "Don't  Know"  is  typically  included  for  all  closed-ended 
questions,  however  this  selection  is  not  provided  to  the  respondent,  only  marked  if  that  is 
what  is  said.  Reading  them  aloud  will  skew  results  significantly. 

•  Attitudinal  Questions  -  typically  asked  on  a  five-point  scale  to  represent  the  range;  very 
negative,  somewhat  negative,  neutral,  somewhat  positive,  very  positive.  Sometimes  the 
neutral  response  is  removed. 


66  •  PUBLIC  PERCEPTION  SURVEY  DEVELOPMENT  AND  ANALYSIS 


DAHB 


CAA-20 15094 


•  Multiple  Responses  -  Allow  respondent  to  choose  several  (usually  up  to  three)  responses 
from  a  given  set  to  answer  a  question. 

•  Agree/Disagree  -  Provide  a  statement,  and  ask  respondents  if  they  agree  with  it. 

•  Open-Ended  Questions  -  Typically  asked  after  an  attitudinal  question,  to  provide  the 
"why."  Respondents  are  free  to  answer  however  they  want,  and  typically,  contractors 
will  bin  common  answers  together  to  simplify  analysis. 

Example  5,  Sample  Question  Series 

Q 1 .  Have  you  heard  of  the  Abu  Sayyaf  Group? 

(Yes,  No) 

Q2.  How  much  of  a  threat  does  the  Abu  Sayyaf  Group  pose  to  your 
community? 

(Great  Deal,  Fair  Amount,  Not  Very  Much,  None) 

Q3.  Why  do  you  say  that? 

(Open-ended) 

The  first  question  is  used  as  a  skip  question,  so  that  only  people  who  have  heard  of  ASG  are 
asked  questions  about  them. 

The  second  question  provides  the  Likert-scale  spectrum  to  measure  the  public’s  sentiment 
towards  the  group. 

The  third  question  provides  insights  as  to  why  the  public  feels  a  certain  way.  A  respondent 
may  say  that  ASG  poses  no  threat  to  their  community  -  that  could  be  because  ASG  is  simply 
not  present  in  their  community,  or  it  could  be  because  ASG  is  present  and  viewed  favorably 
by  the  population.  These  two  reasons  have  very  different  implications  for  the  analysis,  and 
are  impossible  to  extrapolate  from  the  survey  data  otherwise. 


Focus  Group  Discussions 

Analysts  will  sometimes  have  the  contracting  agency  conduct  focus  group  discussions  with  the 
population,  typically  6  to  8  participants  with  a  trained  moderator.  This  can  either  be  a  stand¬ 
alone  survey  mechanism,  or  used  to  augment  the  typical  quantitative  survey.  Focus  groups  offer 
deeper  insights  into  why  people  feel  a  certain  way  on  a  topic.  They  are  usually  organized  around 
one  or  two  specific  areas,  such  as  upcoming  elections  or  the  local  economy,  and  the  moderator 
facilitates  a  fairly  unstructured  discussion  with  the  participants.  The  composition  of  the  focus 
groups  will  be  particular  to  the  region/subject  matter;  sometimes  the  groups  are  split  along 
religious  lines,  sometimes  divided  into  male  and  female  groups.  Focus  groups  are  also  a  way  to 
see  whether  certain  topics  make  people  more  uncomfortable  and  less  willing  to  be  forthright 
about  their  opinion  -  something  that  can  be  used  to  inform  the  development  of  the  quantitative 
questionnaire. 
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Pre-Test 

Particularly  for  a  new  survey,  or  just  new  questions,  one  should  initiate  a  field  test  with  a  small 
sample  size  (25)  to  see  how  the  questions  play  before  the  actual  fieldwork  begins.  Things  to 
look  for;  Is  the  respondent  comfortable  talking  about  the  subject  being  asked?  Is  the  question 
understood  or  is  it  confusing?  Is  there  a  cultural  issue  that  makes  this  question  unsuitable?  The 
field  test  will  isolate  these  questions,  and  others  that  may  provide  non-significant  insights,  i.e., 
100%  of  respondents  answered  "Don't  Know"  to  a  question,  and  allow  the  survey  team  to  adjust, 
delete,  or  add  questions  based  on  field  test  results. 

Oversee  Fieldwork 

When  and  where  possible,  one  should  observe  and  monitor  the  training  of  interviewers,  and 
monitor  the  actual  fieldwork  to  ensure  field  workers  adhere  to  prescribed  standards  and 
methodologies.  However,  consider  the  sensitivity  associated  with  training  observation  and 
fieldwork  monitoring  -  will  a  survey  agency's  association  with  U.S.  personnel  place  their 
employees  at  risk? 


Examole  6,  Social  Sensitivities  in  Questions 

Do  you  own  a  gun? 
vs. 

Is  it  appropriate  for  someone  in  this  neighborhood  to  own  a  gun  to  protect  his  family? 

When  asked  in  the  Southern  Philippine  Public  Perception  Survey,  the  first  question 
above  yielded  a  very  low  positive  response,  which  was  universally  agreed  upon  to  be 
highly  inaccurate.  Filipinos  did  not  want  to  admit  to  personally  owning  a  gun. 
However,  when  asked  in  the  third  person,  as  in  the  second  question  above,  the  response 
rates  were  much  more  in  line  with  what  operational  data  had  analysts  expecting. 


Quality  Check  the  Data 

When  first  receiving  the  raw  data,  run  a  basic  quality  control  to  check  for  inconsistencies  and/or 
anomalies  in  the  data  that  may  point  to  incorrect  coding  or  other  instances  of  the  contractor 
committing  errors  during  the  survey  collection  and  processing  period.  The  quality  assurance 
(QA)/quahty  control  (QC)  will  provide  greater  assurance  that  the  data  collected  is  valid,  offering 
more  legitimacy  to  the  analysis  that  is  based  off  of  it.  Listed  below  are  some  areas  to  focus  on 
for  a  QC  checklist. 
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Example  7,  QA/QC  Questions 

Do  all  SPSS  variable  labels  and  value  labels  in  the  data  set  mateh  the  final 
questionnaire? 

Are  the  marginal  consistent  with  regional/cultural  knowledge/previous  trends? 

Are  crosstabs  logically  consistent? 

Were  fdter  questions  properly  executed  and  coded? 

Do  the  sampling  variables  match  the  pre-fieldwork  sampling  design? 

Does  the  weighted/unweighted  sample  match  a  known  population  distribution? 

Are  lengths  and  timings  of  interviews  consistent  and  within  the  realm  of  normal?  (do 
interviews  last  5  minutes,  or  happen  in  the  middle  of  the  night?) 

Are  there  overlapping  interviews? 

Does  one  interviewer  complete  in  excess  of  10  interviews  per  day?  (The  standard  is 
10,  but  should  be  discussed  with  the  contracting  agency  beforehand) 

Are  there  any  instances  of  teleportation?  (interviewer  concludes  one  interview, 
begins  another  immediately  after  in  a  different  province) 

Are  there  any  duplicate  cases  in  the  dataset? 


Future  waves 

If  the  command  is  planning  on  conducting  subsequent  waves  to  track  trends,  try  to  maintain  as 
much  consistency  as  possible.  Keep  the  exact  wording  of  the  question  and  coding  the  same. 
Maintain  a  merge  file  to  simplify  analysis.  Ensure  the  contractor  complies  with  the  exact 
specifications  on  how  to  code  the  data.  Be  mindful  of  the  timing  of  the  survey,  as  it  relates  to 
current  events.  Is  there  a  desire  to  capture  sentiment  before  and  after  a  presidential  election? 
After  a  major  operation  or  other  significant  event?  Keep  these  in  mind  while  planning  out  the 
timeline  of  conducting  a  survey. 

5.3  Data  Analysis _ 


There  are  several  different  types  of  analysis  that  can  be  used  on  survey  data,  ranging  from  basic 
to  complex.  This  section  walks  through  the  basics  of  data  analysis,  and  briefly  discusses  some  of 
the  more  commonly  used  tools  for  survey  analysis. 

Common  Tools^^ 

•  SPSS.  The  most  common  survey  software  is  SPSS.  SPSS  is  simple  to  learn  and  fairly 
intuitive,  and  has  lots  of  capability  for  survey  analysis.  It  allows  the  development  and 
application  of  weights,  conduct  basic  descriptive  statistics  and  crosstabs,  as  well  as  more 
advanced  analysis  to  include  clustering,  regression,  the  creation  of  new  variables  based 
on  calculations  of  existing  ones,  etc.  The  data  visualization  for  SPSS  is  not  as  flexible 
and  intuitive  as  Microsoft’s  Excel  and  PowerPoint  applications,  and  some  analysts  prefer 
use  of  Tableau  software  to  visualize  survey  and  other  data. 


'9  Note;  No  endorsement  is  stated  or  implied  by  any  references  to  commercial  software 
applications  in  this  handbook. 
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•  Excel.  With  basic  Excel  skills,  one  can  do  most  survey  analysis.  There  is  no  easy  way  to 
apply  weights  to  data  though,  if  necessary.  Pivot  tables  will  enable  easy  erosstabs, 
filtering,  and  frequencies  of  the  data.  Additionally,  Microsoft  Office  is  universal,  so 
there  is  no  need  to  request  speeial  software.  Creating  eharts  in  Exeel  is  simple,  and  they 
are  easily  manipulated  to  adjust,  re-format,  and  combine  into  one  PowerPoint 
presentation  for  stakeholders. 

Basic  Analysis 

•  Erequeneies.  Basie  marginal  information  offers  broad  insights  into  the  perceptions  on  a 
given  topic  for  the  population  as  a  whole. 

•  Crosstabs.  Understand  relationships  between  variables,  particularly  demographics  -  sex, 
ethnicity,  religion,  and  location  are  typieal  subsets  that  tend  to  have  different  attitudes 
towards  subjects.  Beyond  demographics,  crosstabs  are  important  to  understand 
relationships  between  different  attitudinal  questions. 

•  Trends.  Eongitudinal  analysis  provides  stakeholders  with  the  context  of  the  survey 
findings  -  what  was  it  before?  How  has  it  changed?  What  has  been  the  general 
direetionality  over  time? 

•  Correlations.  Quantify  the  degree  to  whieh  two  or  more  factors  show  a  tendency  to  vary 
together.  It  is  useful  to  see  how  closely  responses  to  one  question  follow  responses  to 
another.  Eor  example,  respondents  who  have  strong  distrust  for  the  government  may  also 
have  strong  distrust  for  seeurity  seetor  institutions. 

•  Cluster  Analysis.  Draw  out  groups  of  respondents  whose  response  profiles  are  similar  to 
one  another,  and  identify  what  eommon  eharaeteristics  the  group  has,  and  what  makes  it 
distinct  from  the  rest. 

In  addition  to  various  quantitative  teehniques  for  analyzing  the  data,  the  analyst  should  have 
regional  expertise,  or  at  least  basie  knowledge,  to  understand,  recognize,  and  expeet  eertain 
cleavages  in  the  data.  This  supplementary  eontextual  knowledge  helps  stakeholders  better 
understand  why  the  data  are  a  eertain  way,  and  isolate  the  atypieal  data  points  from  those  that 
would  be  expected,  given  the  population's  eomposition  and  culture. 

When  compiling  survey  analysis  to  present  to  stakeholders,  the  analyst  should  try  and 
ineorporate  other  atmospherie  information,  analysis,  and  findings  wherever  possible  to  validate 
particular  findings  and/or  determine  areas  of  emphasis  that  need  further  analysis  due  to 
confiieting  results. 

5.4  Report  Presentation  and  Delivery 


Tailor  the  analysis  to  the  stakeholders'  needs  and  preferences.  If  they  prefer  less  PowerPoint, 
draft  a  short  narrative  with  minimal  charts  to  capture  the  key  findings  from  the  data.  Highlight 
the  takeaways  that  are  most  relevant  to  their  problem  set.  Minimize  the  use  of  statistieal 
language,  such  as  covariance,  correlation,  statistieally  significant. 

Do  not  attribute  eausation  for  the  way  things  are.  Show  correlation,  overlay  a  timeline  of  current 
events  over  a  trend  of  a  particular  sentiment,  and  offer  potential  explanations,  but  do  not 
definitively  state  "why"  something  is  the  way  it  is. 
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Data  Visualizations 

Use  color  when  it  has  a  meaning,  not  just  for  the  sake  of  using  color.  Do  not  use  it  too  much  to 
make  it  overwhelming  to  the  audience.  Do  not  choose  a  color  for  a  category  that  would  be 
counterintuitive;  e.g.,  red  should  not  mean  support. 

The  figures  below  provide  examples  of  different  visualizations  for  survey  data.  There  are  no 
absolute  rules  for  how  to  present  survey  analysis,  but  the  figures  below  capture  some  examples 
that  have  worked  for  stakeholders.  Know  the  stakeholder's  preferences  in  order  to  create  a 
presentation  that  fits  their  style  best. 


Percentage  of  Respondents  Who  Trust  the  MNLF 


Marawi 

Isabela 

Southern  Balisan 

Sulu 

— 

Figure  20,  Line  chart  -  depict  changes  in  perception  over  time. 


What  is  the  most  serious  probiem  facing  your 
community? 


Access  to  drinkable  water 
Roads 
Unemployment 
Poverty 
Crime/insecurity 
Health 
Expensive  living 
Agriculture 


]  19% 


0%  5%  10%  15%  20% 


Figure  21,  Geospatial  Displays  -  Show  Figure  22,  Clustered  Bars  -  Illustrate  a  rank 
the  distrihution  across  an  area  order  among  several  choices 
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Demographics  of  Survey  Respondents 


□  Male 

□  Female 


□  Muslim 

□  Non-Muslim 


■  Urban 
□  Rural 


Trust  for  Organizations 


MILF  GRP 


Figure  23,  Pie  Charts-  Figure  24,  Stacked  Charts  -  Display  Likert- 

Show  demographic  or  categorical  scale  data  across  categories 

compositions 


5.5  Limitations  of  Surveys _ 

When  using  survey  data  to  inform  analysis,  there  are  several  things  to  keep  in  mind.  As  with  any 
data  set,  there  are  potential  errors  in  the  collection  and  cleaning  processes  to  watch  out  for  and 
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mitigate  as  possible.  Below  are  some  of  the  more  eommon  errors,  and  how  to  limit  the  effeets  of 
them. 

•  Sampling  Error.  Were  enough  people  asked  the  survey?  Was  the  sample  representative 
of  the  entire  population? 

Use  responsible  sampling  methodologies.  Provide  the  margin  of  error  when  displaying 
analysis  to  stakeholders. 

•  Measurement  Error.  Did  respondents  respond  honestly?  Do  the  data  accurately  reflect 
the  sentiment  of  the  population? 

Vet  the  questions  with  focus  groups.  Have  similar  questions  throughout  the  survey  to 
check  for  consistency.  Provide  interviewer  with  questions  on  whether  they  felt  the 
respondent  was  answering  faithfully. 

•  Coverage  Error.  Did  interviewers  go  where  they  were  supposed  to  go? 

Particularly  problematic  with  conflict  areas.  Conduct  regular  site  visits  during  fieldwork, 
collect  status  reports  from  contractors,  check  data  for  instances  of  teleportation  or  other 
inconsistencies. 

•  Other  Errors.  Was  there  malfeasance?  Incorrect  coding?  Failure  to  follow  interview 
protocol?  Compare  data  to  other  sources  for  consistency.  Conduct  thorough  QA/QC 
checks  on  the  data. 

5.6  Existing  Data  Sources/Surveys 


Several  organizations,  both  governmental  and  non-govemmental  are  already  running  surveys, 
most  commonly  at  the  national  level.  Before  undertaking  this  process,  do  research.  Department 
of  State  (DOS)'s  Office  of  Opinion  Research  conducts  surveys  in  almost  every  country.  Center 
for  Army  Analysis  has  access  to  the  Gallup  World  Poll  dataset.  Would  either  of  these  sources 
fulfill  the  requirements?  What  is  the  U.S.  Embassy  collecting?  Are  there  other  open  source  data 
available?  The  Asia  Foundation  and  Social  Weather  Station  conduct  regular  national  and  sub¬ 
national  surveys  in  the  Philippines.  Surveys  are  expensive  and  time-consuming.  Make  sure  the 
information  is  not  already  available  through  another  avenue.  If  there  are  existing  survey 
vehicles,  can  the  analyst  add  questions  to  that  survey  questionnaire  instead  of  beginning  a  new 
effort?  If  there  are  none,  look  around  to  see  what  other  organizations  would  be  interested  in 
survey  data  of  the  specific  areas.  Can  there  be  a  combined  effort?  Look  particularly  to  United 
States  Agency  for  International  Development  (USAID)  or  DOS  counterparts. 

5.7  Summary 


Surveys  are  just  one  data  point  in  understanding  the  operational  environment.  Public  perceptions 
represent  what  people  think,  not  necessarily  the  reality  on  the  ground.  And  given  the  complex 
nature  of  today's  current  operating  environment,  there  is  no  way  to  attribute  specific  actions  to 
changes  in  perception.  In  order  to  get  a  more  complete  picture  of  the  situation,  include  all-source 
information,  including  operational  data;  atmospherics;  Political,  Military,  Economic,  Social, 
Information,  and  Infrastructure  (PMESll)  analysis;  intelligence;  etc. 
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5.8  Further  Reading 


This  is  a  very  basic  primer  on  public  perception  surveys.  See  sources  below  for  more  detailed 
guidance.  The  internet  is  also  a  valuable  resource. 

Survey  Research  Methods.  Fowler,  Floyd  J. 

The  American  Association  for  Public  Opinion  Research  Website  (https://www.aapor.org) 
for  educational  resources,  standard  definitions,  publications. 
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CHAPTER  6  OPERATIONS  ASSESSMENTS 

6.1  Introduction 


A  paradigm  shift  is  underway  in  the  operation  assessment  of  eomplex  environments.  In  the  past, 
an  assessment  produet  was  an  attempt  to  determine  progress  over  the  preeeding  time  period — in 
essenee  a  history  report — that  organizations  eould  share  with  higher  authority.  Today,  an 
assessment  proeess  is  beeoming  an  inereasingly  robust  attempt  to  improve  our  understanding  of 
the  operational  environment  (OE),  whieh  is  informed  by  rigorously  eolleeted  data  given  meaning 
by  appropriate  analyses,  and  the  proeess  is  done  explieitly  for  the  purpose  of  making  operations 
more  effeetive.  Therefore,  assessment  is  now  forward-looking  to  enable  deeision  makers  to  take 
aetions  most  appropriate  to  aehieve  their  desired  outeomes  given  the  eurrent  state  of  the  OE. 

The  definition  of  operation  assessment  proposed  by  the  assessment  eommunity  for  the  next 
revision  of  Joint  Publieation  (JP)  5-0  makes  this  elear.  Operation  assessment  is:  "An 

operational  activity  that  supports  decision  making  by  determining  progress  toward 
accomplishing  a  task,  creating  an  effect,  or  achieving  an  objective  or  end  state  for  the 
purpose  of  making  operations  and  campaigns  more  effective"  (JP  5-0  proposal). 

In  addition  to  the  primary  purpose  of  making  operations  more  effeetive,  assessment  also  has  a 
seeondary  purpose  of  providing  aeeountability  to  higher  authority.  Indeed,  on  many  oeeasions,  a 
formal  assessment,  summarized  in  a  report,  is  required  by  law  or  regulation. 

Assessors  must  also  be  aware  that  eommanders  may  also  use  assessment  data  or  produets  as 
messaging  tools.  This  aetivity  has  implieations  for  the  veraeity  of  assessment  produets  used  in 
this  way  (see  Mushen  and  Sehroden,  (2014)  "Are  We  Winning?  A  Brief  History  of  Military 
Operations  Assessment"  available  through  Google).  It  is  possible  that  a  eommander  will  want  to 
"eherry-piek"  data  or  results  of  analyses  that  support  a  partieular  message. 

6.2  Assessment  Framework 


Assessment  Framework 

Joint  Doctrine  Note  (JDN)  1-15,  Operation  Assessment,  states  that  any  assessment  framework 
consists  of  three  activities:  organize  and  collect  data,  analyze  data,  and  communicate  results  and 
recommendations. 

Organizing  and  collecting  data  includes  specifying  what  data  require  collection  (information 
requirements),  and  details  about  who  will  collect  it,  how  it  will  be  collected,  and  how  the  data 
will  be  stored  to  support  analysis. 

Analyzing  data  consists  of,  in  general,  answering  these  questions: 

•  Where  are  we? 

•  What  happened? 

•  Why  do  we  think  it  happened?  ("we  think"  is  a  hat  tip  to  the  idea  that  we  are  never  sure 
"why") 

•  So  what? 

•  What  are  the  likely  future  opportunities  and  risks? 
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•  What  do  we  need  to  do? 

Analyzing  the  data  is  largely  an  exereise  in  eritieal  thinking.  Mathematieal  teehniques  may  or 
may  not  be  useful  depending  upon  the  nature  and  quality  of  the  data.  Given  the  nature  of  the 
operational  environment  (OE)  in  eomplex  operations  and  the  quality  of  data  eollected, 
quantitative  teehniques  should  not  be  eonsidered  superior  to  eritieal  reasoning.  The 
appropriateness,  usefulness,  and  likely  validity  of  quantitative  teehniques  require  judgment  from 
experieneed  assessors  and  others  with  appropriate  expertise. 

Analyzing  the  data  is  largely  about  trying  to  understand  what  has  oeeurred  and  why,  and 
therefore,  what  the  joint  foree  should  do  next  in  order  to  be  most  effective.  Assessors  cannot 
avoid  discussions  of  causality.  In  practice,  assessors  will  qualitatively  examine  sets  of  indicators 
that  are  related  to  each  other.  They  will  propose  likely,  possibly  multiple,  causal  explanations 
(candidate  hypotheses)  for  the  movement  of  these  indicators.  Then  they  will  pose  additional 
analytic  questions  designed  to  eliminate  some  of  these  hypotheses  and  isolate  the  most  likely 
cause.  They  do  this  by  going  back  into  the  data  in  order  to  re-examine  and  exclude  candidate 
hypotheses. 

Much  of  this  analysis  is  simple  math  and  statistics  or  simple  data  visualization,  enhanced  by  any 
more  sophisticated  quantitative  techniques  that  are  appropriate.  The  overall  process  should  he 
viewed  as  qualitative  in  nature,  simply  because  the  complexity  of  the  OE  defies  precise  and 
complete  measurement — that  is,  some  important  feature  of  the  OE  will  not  be  precisely 
measureable. 

Communicating  results  and  recommendations  includes  packaging  the  pertinent  information 
that  was  identified  during  the  assessment  process  for  sharing  with  decision  makers  and,  if 
required,  for  reporting  to  higher  authority.  This  information  includes  items,  data,  or  results  that 
the  commander  or  other  decision  makers  have  asked  to  see;  answers  to  specific  questions  that  a 
decision  maker  has  posed;  and  other  things  the  staff  has  discovered  during  analysis  that  they 
think  decision  makers  need  to  see. 

It  is  not  necessary  to  prepare  slides  that  show  the  movement  of  every  indicator.  All  indicators 
should  be  examined  by  functional  area  experts  on  the  staff,  but  only  a  few  are  sufficiently 
important  to  occupy  senior  leaders'  time.  Moreover,  it  is  usually  counter-productive  to  attempt 
to  aggregate  several  related  indicators  into  some  sort  of  index  to  reduce  the  amount  of  data 
assessors  attempt  to  show  senior  leaders.  Again,  the  report  or  brief  should  include  only  the 
things  senior  leaders  have  asked  for  and  the  things  that  the  staff  wants  to  bring  to  their  attention. 

It  is  important  to  remember  that  the  packaging  of  information  is  designed  to  look  forward,  to 
inform  decisions  on  future  operations  in  order  to  make  these  operations  most  effective.  The 
assessment  product,  whatever  form  it  takes,  is  not  merely  a  backward-looking  history  report. 

It  is  also  important  to  understand  that  the  assessment  product,  whatever  form  it  takes,  is  not  the 
assessment  itself,  it  is  merely  a  communication  of  the  pertinent  portions  of  the  assessment. 

The  assessment  itself  is  the  improved  understanding  of  the  operational  environment  (OE), 
and  our  place  in  it,  that  we  get  from  a  systematic  examination  of  the  OE  during  the 
assessment  process.  "The  assessment"  is  a  cognitive,  not  a  physical  or  written  product,  that  is 
analogous  to  a  running  staff  estimate.  The  assessment  product — a  brief  or  a  report — is 
essentially  the  executive  summary  of  this  improved  understanding.  (This  is  commonly 
misunderstood.  Erequently,  the  words  "the  assessment"  when  heard  about  the  headquarters  are 
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referring  to  the  report.  This  is  a  eoneeptual  problem.  For  more  information,  email 
adam.p.shilling.civ@mail.mil.) 

JDN  1-15  contains  an  appendix  (Appendix  C)  that  discusses  the  advantages  and  disadvantages  of 
various  means  of  presentation  of  assessment  results.  It  is  relatively  short,  and  the  document  is 
available  through  Google. 

Outcomes 

"Outcomes"  is  a  single  word  that  includes  all  of  the  following:  end  state,  success  criteria, 
termination  criteria,  objectives,  sub-objectives,  effects,  conditions,  tasks  and  purposes,  or  any 
other  words  that  describe  what  we  want  some  portion  of  the  operating  environment  to  look  like 
in  order  for  us  to  say  that  our  operations  are  successful. 

All  operation  assessment  is  contextual  to  the  desired  outcomes  of  the  command  performing  the 
assessment.  Its  purpose  is  to  provide  the  analysis  to  make  operations  more  effective.  Therefore, 
the  questions  (information  requirements)  and  indicators  that  provide  data  for  analysis  are  linked 
explicitly  to  the  set  of  outcomes  the  command  desires. 

In  addition  to  the  data  a  command  collects  for  its  own  use,  higher  authority  may  also  require  a 
command  to  report  on  various  items  that  are  of  interest  to  the  higher  command. 

The  first  duty  of  an  assessor,  when  possible,  is  to  help  the  planners  write  outcome  statements  that 
are  specific.  Failure  to  do  this  will  make  these  outcomes  very  difficult  to  assess,  but  even  more 
importantly,  unclear  outcomes  make  it  very  difficult  for  subordinate  commands  to  take 
effective  action  to  achieve  them.  Unclear  outcome  statements  often  do  not  focus  or  constrain 
subordinates'  behavior,  making  the  desired  outcomes  more  difficult  to  achieve. 

Linking  Outcomes  to  Indicators 

Once  outcomes  are  as  specific  as  possible,  assessors  should  ask  themselves,  about  each  outcome 
statement,  "What  questions  do  we  need  to  answer  to  know  we  are  accomplishing  this?"  The 
resulting  questions  are  information  requirements  (IRs).  This  list  should  be  scrubbed  and  then 
prioritized  to  focus  collection  and  reporting  on  the  information  that  is  most  valuable.  When  any 
IR  is  sufficiently  specific  that  it  can  be  answered  by  empirical  observation,  the  information  that 
answers  that  question  is  an  indicator. 

Adam  Shilling,  PhD,  developed  a  mental  model  for  linking  outcomes  to  indicators  that  can  help 
assessors  identify  information  requirements  and  indicators,  understand  the  relationships  between 
indicators,  and  organize  indicators  for  collection  and  storage. 
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Outcome  Statement 


Ql:  Is  this  outcome  statement  specific  or 
does  it  need  to  be  broken  up? 


Specific 


Broken  up 


_ \j/ _ 

Q2:  "What  questions  do  we  need  to 
answer  to  know  that  we  are  doing  this?" 


List  of  questions 
(information  requirements) 


- 

More  Specific 
Outcome 

Statements 

Q3:  Can  we  answer  this  question 
with  empirical  evidence? 


Yes 


Ask  more  specific 
questions 


Indicators 


Figure  25,  Mental  model  for  linking  outcomes  to  indicators. 

The  model  is  used  by  beginning  at  the  top  with  any  outcome  statement  and  posing  Q 1 .  If  the 
statement  is  judged  to  be  specific  enough,  the  assessor  proceeds  to  Q2.  If  not,  the  assessor  helps 
the  planner  break  the  original  statement  into  more  specific  outcome  statements  that  "add  up"  to 
the  original  outcome.  Each  of  these  is  evaluated  with  Ql,  and  additional  iterations  through  the 
"design  loop,"  until  all  outcome  statements  are  specific  enough  to  proceed.  At  this  point, 
assessors  are  helping  the  planners  with  "design." 

For  each  specific  outcome,  assessors  pose  Q2.  This  generates  a  list  of  questions,  also  called 
information  requirements  (IRs),  that  the  command  needs  to  answer  to  determine  whether  they  are 
progressing  toward  their  desired  outcomes.  These  questions  need  to  be  refined  until  they  are 
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specific  enough  to  answer  with  empirical  observation  (or  the  judgment  of  a  trusted  observer), 
and  prioritized  in  the  face  of  limited  collection  resources.  All  of  these  answers  are  the  set  of 
indicators  that  are  optimal.  Again,  in  the  real  world,  all  of  them  may  not  be  collectable,  so 
assessors  must  prioritize  the  IRs  and  collect  the  most  important  indicators. 

A  word  of  caution:  some  assessors  try  to  skip  the  list  of  questions  (IRs)  portion  of  the  model. 
When  they  do,  the  typically  develop  a  set  of  all  possible  indicators  that  have  bearing  on  the 
desired  outcomes.  This  creates  a  couple  of  problems:  1.  the  list  of  indicators  is  too  long,  lacks 
focus,  and  is  hard  to  prioritize;  2.  assessors  do  not  understand  the  relationship  of  indicators  to 
each  other.  The  formal  step  of  listing  IRs  (questions)  and  then  identifying  indicators  that  answer 
them  helps  assessors  understand  the  relationship  between  several  indicators  that  may  answer  a 
single  question.  Also,  most  people  find  the  list  of  questions  easier  to  prioritize  than  an 
exhaustive  list  of  potential  indicators.  In  essence,  the  formality  of  posing  the  questions  focuses 
the  identification  of  indicators. 

Some  observations  from  experience:  Sometimes  several  questions  (IRs)  may  be  answered  or 
informed  by  the  same  indicator  or  set  of  indicators.  Some  questions  may  be  answered 
individually  by  one  or  more  indicators;  or  several  related  questions  may  be  answered  collectively 
by  a  set  of  indicators.  Some  indicators  are  straight  forward  and  observable,  e.g.,  the  number  of 
recruits  that  qualified  with  their  individual  weapons  in  the  current  class  of  boot  camp.  Others 
will  require  a  judgment,  which  will  require  an  observer(s)  trusted  to  make  the  judgment,  e.g.,  a 
platoon  is  proficient  in  the  task  "react  to  ambush."  Many  more  require  observers  to  try  to 
classify  an  empirical  event  according  to  set  of  criteria,  e.g.,  was  this  attack  simple  or  complex? 

Some  issues  will  require  a  set  of  IRs  and  indicators  to  give  the  command  a  thorough 
understanding  of  each  issue.  Others  may  require  only  enough  information  to  determine  that  an 
issue  is  or  is  not  a  problem  for  the  command.  If  a  problem  exists,  the  command  can  focus 
additional  effort  and  resources  on  understanding  and  resolving  the  issue.  If  not,  the  command 
may  be  content  with  merely  monitoring  it. 

Fully  specifying  indicators 

Selected  indicators  must  be  sufficiently  well-specified  such  that  any  one  indicator  can  be 
collected  consistently  by  multiple  observers — at  different  places  or  over  time.  Each  will  need  a 
definition,  a  plan  for  collecting  the  data  (who,  what,  when,  why,  and  how),  and  be  sensitive  to 
change  within  a  relevant  time  frame.  If  it  is  calculated,  it  must  have  a  formula;  and  it  may  have  a 
target  or  threshold  or  a  desired  rate  of  change. 

Among  other  pending  changes  in  U.S.  doctrine  in  2015,  the  definitions  of  common  terms 
associated  with  a  regime  of  metrics  or  indicators  are  pending  change.  Joint  Doctrine  Note  (JDN) 
1-15,  Operation  Assessment,  proposes  the  following  new  definitions  to  replace  those  currently  in 
JP  3-0: 

Measure  of  effectiveness  (MOE):  An  indicator  used  to  measure  a  current  system  state,  with 
change  indicated  by  comparing  multiple  observations  over  time. 

Measure  of  performance  (MOP):  An  indicator  used  to  measure  a  friendly  action  that  is  tied  to 
measuring  task  accomplishment. 

Indicator:  A  specific  piece  of  information  that  shows  the  condition,  state,  or  existence  of 
something,  and  provides  a  reliable  means  to  measure  performance  or  effectiveness. 
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Under  previous  definitions,  a  MOE  was  a  eriterion  used  to  assess  ehanges  in  systems,  and  a 
MOP  was  a  eriterion  tied  to  measuring  task  aeeomplishment.  An  indieator  provided  insight  into 
a  MOE  or  MOP.  These  previous  definitions  ereated  a  hierarchy  in  which  MOEs  and  MOPs  were 
informed  by  indicators.  This  old  metric  regime,  in  practice,  created  a  number  of  problems  with 
the  organization  of  information  for  collection  and  analysis,  including  that  MOEs,  as  written, 
were  rarely  specific  enough  to  measure  anything.  The  new  definitions  destroy  this  hierarchy  and 
provide  that  both  MOEs  and  MOPs  are  types  of  indicator  that,  by  definition,  must  be  specific. 

Sources  for  collecting  indicators  include  routine  reporting,  intelligence  and  operations  reporting, 
surveys,  interagency  or  international  partners,  host  nation  reporting,  or  sometimes  special  data 
collection  efforts  for  data  not  available  elsewhere.  Erequently,  external  organizations  may 
collect  data  that  are  of  higher  quality  on  some  topic  than  joint  forces  can  collect  themselves,  e.g., 
civilian  casualty  data  in  Afghanistan  are  better  collected  by  the  United  Nations  than  by  ISAE. 
Also,  assessors  will  want  to  check  military  data  against  data  owned  by  external  organizations, 
both  to  double  check  military  collection  and  to  be  able  to  explain  why  military  data  may  vary 
from  other  data  published  in  international  media. 

6.3  More  on  Causality 


Data  scientists  frequently  shy  away  from  talking  about  causality  because  the  data,  if  they  are  of 
the  right  type,  can  only  demonstrate  a  statistical  correlation  of  a  variable  (indicator)  to  others.  In 
assessment  however,  assessors  cannot  avoid  the  issue  of  causality  because  ultimately,  assessors 
and  subject  matter  experts  on  the  staff  must  explain  why  indicators  are  moving  as  they  are.  Any 

operational  plan  that  orders  a  set  of  activities  in  pursuit  of  a  set  of  outcomes  is  a  set  of 
causal  hypotheses — that  these  activities  will  cause  the  changes  in  the  OE  that  are  specified 
as  outcomes.  Assessment  can  be  viewed  as  a  testing  of  these  hypotheses.  Therefore,  assessors 
must  address  issues  of  causality. 

Assessors  cannot  assume,  however,  that  since  an  outcome  occurred,  therefore  friendly  action 
caused  it.  The  OE  is  full  of  other  actors,  some  allied  or  sympathetic  to  US  goals,  some  opposed 
or  adversarial  toward  American  goals,  and  many  third  party  actors  that  are  pursuing  agendas  of 
their  own. 

Assessors  need  to  consider  sets  of  related  indicators  together,  determine  possible  causal 
relationships  that  would  account  for  the  movements  in  indicators,  and  then  go  back  into  the  data 
in  an  attempt  eliminate  some  possible  causes,  isolating  the  most  likely. 

6.4  Alternatives  to  MOE-MOP  Constructs 


Some  assessment  problems  can  be  addressed  through  methods  other  than  the  formal  monitoring 
of  specific  MOEs  and  MOPs.  Some  alternatives  include: 

The  use  of  published  standards,  such  as  Training  and  Evaluation  Outlines  (T&EOs)  and  Soldiers' 
Manuals,  provide  "performance  steps"  that  describe  how  to  correctly  perform  collective  and 
individual  military  tasks.  These  can  be  used  as  training  references,  as  references  for  the 
evaluation  of  training,  and  as  references  for  the  assessment  of  operations.  The  correct 
performance  of  tasks  will  lead  to  more  effective  individuals  and  units.  Moreover,  if  an  outcome 
is  not  achieved,  but  we  know  from  a  task  assessment  that  the  task  was  performed  correctly,  then 
we  know  that  performance  of  that  task  does  not  lead  directly  to  that  outcome.  If  the  task  were 
performed  incorrectly,  we  might  conclude  the  task  should  be  performed  again. 
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Inspection  programs  usually  feature  a  qualified  observer  armed  with  a  checklist.  The  observer 
answers  those  checklist  questions  about  the  performance  or  effectiveness  of  a  unit,  which  is  an 
assessment  of  the  unit's  capability.  He  may  also  mentor  the  unit  on  how  to  do  better  against  the 
items  on  the  checklist,  and  finally,  the  inspection  itself  is  an  incentive  for  the  evaluated  unit  to 
periodically  examine  itself  and  ensure  it  is  in  compliance  with  standards. 

Reports  the  headquarters  receives  are  assessment  documents.  Ultimately,  assessment  data  do  not 
exist  in  a  sterile  database  for  assessors'  analytic  use.  Observers  in  the  field  must  witness  and 
report  the  items  that  are  of  interest  to  assessors,  and  therefore,  assessors  must  tell  them  what  to 
report.  Many  items  are  reported  routinely  already,  and  gathering  that  data  is  a  matter  of  gleaning 
them  from  existing  reporting.  For  items  not  currently  reported,  assessors  and  staff  members  will 
need  to  construct  a  report  format  that  enables  disparate  observers  to  consistently  report  the 
required  information.  For  example,  if  a  headquarters  is  tracking  naval  incursions  of  an 
adversarial  navy  into  a  friendly  navy's  waters,  routine  spot  reports  will  provide  size,  activity, 
location,  unit,  time,  and  equipment  information  that,  when  analyzed,  will  answer  many  of  the 
headquarters'  questions  about  the  adversary's  activity  trends  and  likely  intentions. 

Finally,  when  something  out  of  the  ordinary  occurs,  assessors  might  need  to  determine  what 
happened  and  why  to  either  ensure  it  never  happens  again  (for  a  negative  event)  or  to  institute  it 
as  a  best  practice  (for  a  positive  event).  For  example,  mission  personnel  returning  from  multi¬ 
day  missions  in  the  Horn  of  Africa  reported  during  their  after  action  reviews  that  they  had 
difficulty  contacting  the  Joint  Operations  Center  (JOC)  by  phone  for  submitting  evening 
accountability  reports.  The  JOC's  representative  to  the  after  action  review  panel  investigated  and 
determined  the  failure  of  the  JOC  watch  to  answer  the  phones  was  due  to  the  fact  that  phone 
ringers  were  turned  off  during  the  daily  afternoon  brief  and  not  consistently  turned  back  on.  He 
instituted  corrective  measures,  and  reports  of  the  JOC's  failure  to  answer  the  phones  dropped  to 
zero. 

6.5  More  on  Assessment  Products 


Usually,  the  words,  "the  assessment,"  heard  about  the  headquarters  refer  to  the  assessment 
product — usually  a  report  or  slideshow.  In  many  staff  actions,  a  focus  on  the  "deliverable"  is 
helpful  to  get  work  done.  In  operation  assessment,  this  attitude  is  not  as  useful.  Or  more 
accurately,  the  "deliverable"  for  assessment  is  an  effective  operation,  the  "product,"  whatever  it 
is,  is  merely  a  communication  of  the  pertinent  portions  of  the  assessment — the  unwritten 
cognitive  product — for  informing  senior  leaders  or  providing  accountability. 

Assessment  products  in  the  past  (and  often  currently)  have  not  reflected  this  sensibility.  Often 
assessment  briefs  have  consisted  of  many  slides  that  each  track  an  indicator.  This  is  not  a 
problem  in  itself  and  tracking  indicators  is  critical,  but  only  a  few  indicators  are  sufficiently 
important  that  a  senior  leader  will  need  to  see  a  slide  of  its  trend.  One  important  example  was 
the  end  strength  of  the  Afghan  National  Army  (see  below).  Four-star  leadership  and  higher 
authority  wanted  to  see  this  trend  because  it  was  critical  to  U.S.  perception  of  its  success  in 
building  Afghan  institutions.  A  less  important  example  was  the  trend  of  the  percentages  of 
Afghan  army  recruits  qualifying  with  their  individual  weapons.  The  latter  metric  is  critical  to  the 
master  sergeant  who  runs  the  rifle  range  and  to  the  lieutenant  colonel  who  commands  basic 
training,  but  it  is  not  something  general  officers  need  to  see. 
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ANSF  Endstrength  On  Track 

N/m) 

to  Meet  October  2012  Goals  (2/3) 

OTAN 

ANAEndstrength  Growth,  Goals,  and  Forecasts 


Figure  26,  Assessment  product  example  1, 

Many  past  products  also  consisted  of  a  simple,  often  bulleted,  summary  of  a  "line  of  effort." 
Many  of  these  have  a  stoplight  depiction  that  communicates  that  the  command  is  on  track  to 
achieve  the  outcomes  of  this  line  of  effort  and  a  depiction  of  the  trend — are  we  moving  in  the 
right  direction  (see  below)?  These  depictions  are  simplistic.  They  may  be  sufficient  as 
communication  tools  provided  they  stimulate  the  right  discussion  among  senior  leaders,  generate 
appropriate  recommendations  from  the  staff,  and  action  is  taken  on  them.  However,  this 
depiction  cannot  be  mistaken  for  "the  assessment"  because  the  staff  must  still  chum  through  the 
indicators,  determine  what  they  mean,  posit  causal  relationships,  and  make  recommendations  for 
the  command.  The  staffs  mere  production  of  this  slide  does  not  constitute  assessment. 
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APR  12 
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A  A  A  AA 


N/A 


2.1a:  Afghan  Army  and  Polica  capable  of  shouldering  additional  security  tasks 
with  leu  assistance  from  ISAP  in  Tranche  1  areas.  (TPOR,  AMNODNE) 

2.1. b:  Afghan  Army  and  PolicecapaUe  of  shoulderingadditiondlsecurttytasks  with  less 
assistance  from  ISAF  In  Tranche  2  areas. 

2.2.  APPF  capable  of  replacing  contracted  security  for  development,  1C  .and  ISAF.  (CIS) 

2.3.  ANSF  force  structure  pctMh  goals  achieved  and  maintained(reauiting/attrition). 
(NTM-A,  CF50CX-A) 

2.4.  Ministries  (MOI/MOO)  capable  of  effectively  and  efficientiy  managing  the  ANSF.  (NTk^*A] 


NATO 
NT  01  an 


2014  Camoakin  Oblective 

A  sustainable  ANSF  leading  In  population 
security  and  law  enforcement,  accountable  to 
and  serving  the  Afghan  people.  _ 

wv 


OnTra«k 
Tam  port  nr 


On  Track 
Enduring 


SciB 

AdikveA 


a 


V. 


Apr  2011 


V. 


JullOll 


f  >  Oct  2011 


N/A 

N/A 


Note:  Most  kdermediatecainpalcn  Objectives  are 
both  new  and  more  challenging  than  Jul  2011. 

2.5.  Afghan  Army  and  Police  leaders  at  a  level  capable  of  conduebng  independent  security  operations  with  essistance.  (DC) 

2.6.  Afghan  Army  and  Police  trusted  by  the  population  to  maintain  security  and  enforce  Afghan  law.  (ANQAR,  FOGHORN) 

2.7  MOI,  MOO,  and  GFC  capable  of  proyidlne  operaBonal  sustainment  in  areas  where  transition  Is  complete. 


1. a.  Is  the  ANSF  capable  in  Tranche  1  areas?  —  Mostly  Positive  Q| 

Tranche  1  transition  security  rated  "good"  or  "very  good"  in  August  TPOR,  incl.  Lashkar  Gah  &  Mehtar  Lam 

-  Tranche  1  area  violence  since  Jun  2011  is  lower  than  2010;  ANSF  led  responses  to  the  2  major  Kabul  attacks 

2.  Is  APPF  capable  of  replacing  contracted  security  for  development,  1C,  and  ISAF?  —  Negative  13 

-  GIRoA  has  not  formed  a  state  owned  enterprise  or  business  plan  to  execute  APPF  missions;  25k  req.  (dev,  ISAF) 

-  APPF  force  of  7,000  is  growing,  but  connmitted  to  GIRoA  and  priority  private  sector  requirements  thru  Mar  2012 

3.  Is  ANSF  growth  on  track?  —  Mixed 

-  ANA  and  ANP  end  strength  on  track  re  Oct  2012  targets  (ANP  157k,  ANA  195  k);  ALP  grew  25%  to  8,384 
But  MoD 's  ANA  new-recruit  and  training  attrition  reporting  is  inconsistent  and  inaccurate 

4.  Are  Ministries  Able  to  Manage  the  ANSF?  —  Mixed  ]■ 

Ministerial  department  capability  milestone  (CM)  achievement  trend  remains  positive 

-  Projected  CM-IB  (req.  coal,  overs't)  dates  for  40  depts.  slipped  right;  civil  serv.  reform  Is  req.  to  stop  backsliding 

5.  Are  ANSF  leaders  capable  of  conducting  independent  security  operations  with  assistance?  -  Mbced  D| 

Most  units  forecasted  Indep.  In  2013;  but  17%  of  ANA  Kdks,  OCCs,  and  priority  ANP  elements  not  ngw  assessed 

6.  Is  the  ANSF  trusted  by  the  population  to  maintain  security  and  enforce  Afghan  law?  —Positive  M 

ANQAR  &  FOGFIORN  surveys:  Growing  majority  of  Afghans  say  ANSF  is  capable,  well  trained,  and  fair  ^ 


Figure  27,  Assessment  product  example  2. 

Assessment  reports  frequently  feature  narrative  descriptions  of  happenings  and  their  meaning 
that  are  illustrated  by  appropriate  graphics.  A  major  example  is  the  "Report  on  Progress  Toward 
Security  and  Stability  in  Afghanistan,"  also  known  as  the  "1230  reporf'^o  (available  through 
Google).  This  report  provides  accountability  to  the  U.S.  Congress  on  military  activities  in 
Afghanistan.  It  is  a  large  effort  to  describe  the  results  of  assessment  analysis,  but  it  is  also  a 
model  of  a  narrative-based  assessment  communication  tool  that  can  be  scaled  down  for  use 
within  a  headquarters.  Frequently,  an  effective  depiction  may  be  a  chart  or  several  charts  that  are 
captioned  with  a  narrative  explaining  what  they  mean,  what  causal  relationships  exist,  and  what 
the  command  should  do. 

The  bottom  line  of  this  discussion  is  that  assessment  products  may  take  a  variety  of  forms,  some 
of  which  may  or  may  not  be  useful,  and  so  the  product  format  must  be  evaluated  according  to 
this  standard;  Does  the  preparation  of  this  depiction  provoke  the  conversation  among  the  staff 
that  promotes  nuanced  understanding  of  our  OE  and  help  us  select  appropriate  actions  for  our 
forces?  Does  this  depiction  allow  the  effective  communication  of  our  findings  and 


20  In  2015  the  1230  Report  was  re-codified  by  Congress  to  section  1225  of  the  US  National 
Defense  Appropriation  Act  (NDAA)  and  now  known  as  the  1225  Report,  along  with  a 
classified  annex. 
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recommendations,  and  their  implications,  to  senior  leaders?  If  so,  the  assessment  product  is 
probably  successful. 

One  more  thing:  In  general,  CAA  recommends  showing  senior  leaders  only  what  they  ask  to  see 
and  what  the  staff  thinks  they  need  to  see.  The  former  is  highly  correlated  with  the  information 
that  is  really  important,  and  the  latter  are  the  things  the  staff  discovers  during  the  assessment 
process  that  need  to  be  brought  to  the  attention  of  senior  leaders. 

6.6  To  read  more 


Start  with  Joint  Doctrine  Note  (JDN)  1-15,  Operation  Assessment  (2015)  (also  available  through 
Google).  At  the  time  of  this  writing,  it  is  the  most  advanced  published  work  on  assessment 
doctrine.  Older  sources  are  useful  for  perspective,  and  there  are  number  of  people  performing 
assessment  in  a  manner  consistent  with  older  publications — you  may  work  for  one  of  these 
people  soon — ^but  JDN  1-15  is  paradigm  shift  in  why  the  joint  force  does  operation  assessment, 
and  older  publications  reflect  obsolete  ways  of  thinking  about  assessment  processes. 

CAA  also  recommends  several  instructive  critiques  of  the  old  ways  of  assessment.  These  are: 

Jonathon  Schroden,  Why  Operations  Assessments  Fail  (Naval  War  College  Review,  Autumn 

2011). 

Stephen  Downes-Martin,  Operations  Assessment  in  Afghanistan  is  Broken  (Naval  War  College 
Review,  Autumn  2011). 

Ben  Connahle,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency 
(RAND,  2012). 
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CHAPTER  7  GEOSPATIAL  ANALYSIS 

7.1  Introduction 


Definition.  Geospatial  analysis  seeks  to  find  correlation  between  physical,  cultural,  or 
operational  features,  to  answer  a  specific  question.  Typically,  a  Geographic  Information  System 
(GIS)  or  geospatial  software  is  used  to  apply  statistical  and  proximal  techniques  to 
geographically  referenced  data  to  determine  if  relationships  exist  within  and/or  between  datasets 
(map  layers)  to  provide  insight  to  the  study  problem.  Simply  put,  geospatial  analysis  is  the  use 
of  geographic  data  to  increase  the  understanding  of  a  given  phenomenon.  The  results  can  be 
presented  in  a  map,  graphic,  or  in  tabular  form  depending  on  the  nature  of  the  analysis. 

The  Purpose  of  Geospatial  Analysis.  Geographic  analysis  is  the  "where"  in  the  analyst  query. 
’Where"  an  event  occurs  is  just  as  important  as  "when"  and  "how"  in  the  context  of  discerning 
"why."  Used  appropriately,  geospatial  analysis  can  summarize  large  quantities  of  data  clearly 
and  concisely.  The  addition  of  a  map  to  operational  analysis  will  provide  a  reference  framework 
to  the  analysis  to  enhance  and  accelerate  its  understanding.  This  chapter  includes  several 
examples  to  show  the  advantages  of  this  technique. 

7.2  Geospatial  Software  and  Its  Uses 


Geospatial  analysis  is  a  powerful  tool  that  complements  other  types  of  analyses.  There  are 
several  geospatial  software  packages  currently  used  by  ORSA  analysts,  to  include  (1)  ArcGlS, 
(2)  FalconView,  and  (3)  Web-Enabled  Temporal  Analysis  System  (WebTAS;  commonly  used 
with  the  Combined  Information  Data  Network  Exchange  [CIDNE]  database).  These  tools  plot 
geo-referenced  data  to  reveal  geospatial  relationships  within  and/or  between  the  datasets,  as  well 
as  other  features  such  as  populated  places,  administrative  boundaries,  roads,  or  operating  areas. 
Each  of  the  aforementioned  software  programs  has  different  capabilities  and  strengths.  One  can 
use  any  combination  of  them  to  conduct  geospatial  analysis  effectively.  ArcGlS  and  WebTAS 
are  able  to  plot  densities,  conduct  clustering  of  specific  incidents,  determine  distance/  direction, 
and  display  results.  WebTAS  is  able  to  create  a  "video"  of  incidents;  showing  the  incident 
pattern  both  spatially  and  over  time. 

7.3  Examples  of  Geospatial  Analysis 


Geospatial  analysis  has  played  an  important  role  in  analyses  and  assessments  in  Iraq, 
Afghanistan,  Philippines,  the  Horn  of  Africa,  and  elsewhere.  Common  themes  used  in-theater 
include  density  of  attacks,  effectiveness  of  attacks,  operating  areas,  area  assessments  (security, 
governance,  development,  and  overall),  MEDEVAC  response  times,  and  polling  data 
representation.  Geospatial  analysis  often  compresses  complex  issues  to  enrich  the  commander's 
situational  awareness.  The  following  are  examples  of  real-world  geospatial  analysis: 


DAHB 


GEOSPATIAL  ANALYSIS  •  85 


CAA-20 15094 


Figure  28,  Napoleon's  march  to  Moscow  in  1812 

Minard's  use  of  geospatial  analysis  in  1869  to  portray  Napoleon's  1812  Russian  campaign  is 
generally  referred  to  as  the  best  statistical  graphic  ever  drawn  (see  Figure  28  above).  Minard 
uses  six  separate  variables:  (1)  latitude  and  (2)  longitude  of  the  army  as  it  moved;  (3)  line  width 
continuously  marks  the  size  of  the  army;  (4)  lines  showing  the  directions  the  army  travels  (5) 
temperatures  on  the  return  trip;  and  (6)  dates  (time).  The  variables  were  each  instrumental  in  the 
force's  failure.  Together,  Minard  uses  these  variables  to  tell  the  story  of  Napoleon's  force- 
attrition  over  the  6  months  of  the  campaign. 
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MEDEVAC  Coverage  (Notional) 
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Figure  29.  MEDEVAC  coverage 

Figure  29  depicts  medical  treatment  facilities,  aircraft  bases,  MEDEVAC  events,  and  response 
bands.  This  illustration  is  especially  useful  to  a  planner  who  is  considering  a  large  operation 
requiring  significant  MEDEVAC  capability.  By  identifying  potential  areas  of  inadequate 
coverage,  the  planner  can  work  with  MEDEVAC  units  to  reposition  assets  for  the  operation  in  a 
way  that  ensures  timely  MEDEVAC. 
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Figure  30.  District  assessments  in  Afghanistan 

Figure  30  shows  the  overall  distriet  ratings  on  a  eountry  map.  This  type  of  map  gives  the 
eommander  a  quick  snapshot  of  the  general  situation  to  assist  in  identifying  successes  and  areas 
of  concern.  While  a  map  does  not  explain  the  entire  situation,  it  does  provide  the  commander 
with  a  clear  picture  of  the  current  assessment  in  his  or  her  area  of  responsibility  (AOR).  It  is 
highly  recommended  that  more  detailed,  narrative  assessments  are  completed  and  briefed  to  the 
commander  in  addition  to  the  map. 
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Bangsamoro  Perception  Survey  (BPS)  Sampled  Barangay  with  the 
Number  of  Times  Each  Barangay  Participated  in  Southern  Philippines 
Public  Perception  Survey  (SPPPS),  Waves  1  through  8,  November  2014 


Province  boundary 

-  Municipality/  City  boundary 

Barangay  boundary 

BPS  sampled 
barangay 


Times  Participated 
in  SPPPS  Waves 

□  o 

□  1 

□  2-3 

1:34-5 


0  5.5  11  16.5  Kilometers 

\ - 1 - >  I 


Boundary  representation  is 

not  necessarily  authoritative.  CAA/  gmc  1547  09-2015 


Figure  31.  Map  of  areas  polled  in  the  Philippines 

Figure  3 1  highlights  the  Zamboanga  Peninsula  that  contractors  polled  for  the  BPS  and  relative 
participation  in  the  SPPPS.  This  map  highlights  the  Barangay  suitable  for  further  analysis,  given 
the  number  of  survey  results  available. 

7.4  Summary _ 


Geospatial  analysis  is  a  supplementary  tool  for  the  deployed  ORSA  analyst's  toolkit.  Geospatial 
software  can  summarize  large  quantities  of  geographically  referenced  data  quickly,  clearly,  and 
concisely.  Geospatial  analysis  opens  a  new  area  to  explore  and  explain  the  relationships  and 
correlations  within  and/or  between  multiple  geographic  datasets.  The  examples  used  here  are 
only  a  subset  of  the  range  of  applications. 
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For  training  reservations  contact  the  National  Geospatial  Agency  (NGA)  College  Registrar  (571- 
588-2968)  for  the  free  1-week  courses  "Fundamentals  of  Geographic  Information  Systems  and 
Intermediate  Geographic  Information  Systems  (GIS)  for  Analysis."  These  courses  are  available 
in  the  St  Louis,  MO  and  Northern  Virginia  areas.  For  additional  examples  or  further  assistance, 
contact  NGA's  Geospatial  Intelligence  section  or  the  CAA  Operations  Analysis  (OA)  reachback 
team  POCs  listed  at  http://www.caa.armv.mil/Commanders  Trifold  Jan2016.pdf. 
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CHAPTER  8  REACHBACK  CAPABILITIES 

8.1  Why  Reachback? 


The  Generating  Force,  i.e.,  institutional  army  analytical  organizations,  provide  analyses  through 
an  integrated  combination  of  embedded  ORSA  analysts — either  assigned  or  attached  to  Army, 
Joint,  and  Multi-National  headquarters — and  through  analytical  reachback  support  (also  known 
simply  as  "reachback").  Operations  Research/Systems  Analysis  is  vital  in  reducing  commanders' 
uncertainty  through  sound  reasoning  and  well-constructed  analytical  models.  Reachback  allows 
ORSA  analysts  serving  at  Division,  Corps,  ASCCs,  and  Multi-National  Command  headquarters 
in  deployed  environments  to  draw  upon  resources  and  capabilities  of  Generating  Force 
organizations  and  institutions  whose  primary  mission  is  to  generate  and  sustain  the  Operational 
Army's  capabilities  for  employment  by  Joint  Force  commanders. 

As  a  deployed  ORSA  analyst,  one  applies  logical  reasoning  and  sound  processes  to  solve  highly 
complex  problems  at  the  operational  and  strategic  levels  when  no  readily  apparent  solutions 
exist.  One  applies  analytical  methods  to  perform  trade-off  analysis,  compare  courses  of  action, 
allocate  critical  resources,  and  assess  operational  effectiveness.  These  methods  are  an  integral 
part  of  Army  and  Joint  leadership  decision-making  processes  to  organize,  staff,  train,  equip, 
sustain,  and  resource  current  and  future  forces.  One  conducts  analyses  using  one's  personal 
talents  and  skill  set  unless  the  complexity  and  scope  of  a  given  problem  require  additional 
resources. 

8.2  Motivational  Factors  for  Reachback 


Integration  of  Analytical  Effort.  Reachback  connects  Operational  Army  analytical  efforts  to 
Generating  Force  organizations.  This  connectivity  ensures  analytical  relevance  along  with 
greater  capability  within  the  operational  headquarters.  Specifically,  reachback  provides  visibility 
of  critical  analytical  requirements  to  Generating  Force  organizations.  Additionally,  this 
connectivity  captures  institutional  knowledge  for  future  deploying  ORSA  analysts. 

Subject  Matter  Expertise.  Generating  Force  organizations  possess  a  wide  array  of  expertise. 
Given  the  limited  number  of  deployed  ORSA  analysts  and  the  complexity  of  emerging  issues, 
reachback  provides  a  valuable  mechanism  for  expanding  one's  expertise  and  analytical 
capabilities.  Furthermore,  organizations  that  provide  reachback  are  frequently  staffed  with 
ORSA  analysts  who  have  or  will  deploy.  This  provides  important  continuity  for  analytical 
support  to  theater  commanders. 

Additional  Data.  Reachback  expands  available  data  and  offers  contextual  reference  for  theater- 
specific  analytical  problems. 

Analytical  Review  Process.  Reachback  provides  a  natural  review  process  for  analytical  efforts. 
Through  collaborative  reachback  and  Generating  Force  organizational  capabilities,  multiple 
personnel  and  organizations  review  reachback  analytical  products.  While  the  review  process 
requires  more  time,  it  ensures  high  quality  products. 

Analytical  Tools.  Reachback  provides  one  with  access  to  sophisticated  tools  and  additional 
expertise.  Two  examples  of  support  provided  to  operational  headquarters  are  provided  below. 
Figure  32  is  an  example  of  an  R  script  written  to  replace  a  manual  process  used  downrange  to 
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download  and  index  information  from  web  pages  on  the  internet,  which  saved  hundreds  of  hours 
of  manual  processing  time  for  a  deployed  headquarters. 


^  NGramWorking.R*  •:  ^TrumpPressReleaseCode.R*  *  ^  googIc.plus.ertrartor.R*  ••• 

Q  [G  Source  on  Save  ^  ’  L* 

1  #####  Follow  the  5  steps  below  to  process  the  web  page  docunients|,  extracting  a  table  of  names  and  numbers. 

2 

3  #####  Step  0:  highlight  the  below  2  lines  and  click  "run". 

4  #####  After  running  once  on  a  new  machine,  make  the  lines  below  look  like:  #install.packages(’XML') 

5  #####  This  will  prevent  R  from  re-installing  the  packages  each  time. 

6 

7  i nstal 1. packages (‘ XML ’ ,  dependencies=TRUE) 

8  install. packagesCstringr’ ,  dependencies=TRUE) 

9 

10  #####  Step  1:  save  the  website/html  file  in  a  folder  that  you  can  find.***save  as  a  text  (.txt)  file!!! 

11  #####  Make  sure  only  html  text  is  in  the  file.  The  file  name  should  look  like:  iny_read_file_name.txt 

12 

13  #####  Step  2:  Change  the  below  to  match  the  folder  you  chose  to  save  the  file  in 

14 

15  dir<-  "C:/users/samuel. h. huddleston/Documents/Projects/BahniQuickTurn/Test’' 

16 

17  #####  Step  3:  Change  the  below  to  match  the  website/html  file  name  in  .txt  format 

18  read. file  <-  "my_read_file" 

19 

20  ♦####  Step  4:  change  the  below  to  make  a  new  name  for  the  output  file  (excel  table) 

21  write. file<-  "my_write_file" 

22 

23  #####  Step  5:  Select  everything  on  this  page  (Ctrl-A)  including  everything  here  and  run  (ctrl-R  on  a  windows 

24  #####  machine  or  ctrl-Enter  on  a  mac  machine) 

25  *####  Note  that  the  two  "install. packages"  lines  should  have  been  run  once,  and  then  conmented  out  prior  to 

26  #####  executing  this  step. 

27 

28  library(XML) 

29  library(stringr) 

30 

31  setwd(dir) 

32  X  <-  htmlparse(paste(read. file, ’.txt' ,sep=")) 

33 

34  website  <-  xpathSApply(x,  '7/a/®href") 

35  website. process  <-  gsub("\\D","",as.character(website)) 

36  num  <-  unique(website. process) 

37 

38  name  <-  as. character(xpathSApply(x,  "//®alt")) 

39 

40  df  data.frameC'Number"  =  paste0(’"= . ,num,  . '"),  "Name"  =  name,  stringsAsFactors=FALSE) 

41 

42  write. csv(df,  paste(write.file,Sys.Date(), '.csv' ,sep=’ '),  row.names  =  F,  quote  =  F) 

43 


1:65  O  (Top  Level)  ? 


_4Run  ^ 


R  Script  i 


ft' 

i  * 

f 

41  "I 
~ 

Clpi 

41  51 

41  Cl 


P 

li 

ti 

w 

0 

a 

n 

s 

fi 

a 

a 

n 

n 

r: 


gni 


Figure  32,  Example  code  (script)  written  to  support  reachback  request 

Figure  33  is  an  example  of  an  interaetive  web-based  analytie  tool  developed  for  an  Army 
operational  headquarters  to  provide  time  series  analysis  and  visualization  of  enemy  activity. 
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r  ^ 


c*  . .  .  ^  ; 


Figure  33.  Example  web  application  developed  to  support  reachback  request 

Required  Surge  Capability.  The  connection  between  Operational  Army  commands  and 
Generating  Force  organizations,  enabled  by  reachback,  allows  for  analytical  surge  capability  as 
required.  For  example,  from  the  onset  of  OIF  and  OFF,  CAA,  TRAC,  Army  Materiel  Systems 
Analysis  Activity  (AMSAA),  and  the  United  States  Military  Academy  (USMA)  provided 
Deployed  Analyst  Support  Teams  (DASTs)  to  ASCCs  in  Iraq  and  Afghanistan.  A  recent  change 
in  U.S.  military  operational  focus  has  resulted  in  these  organizations  expanding  their  analytical 
support  to  commands  located  in  additional  geographic  locations,  e.g.,  the  Philippines  and  Africa. 

8.3  Reachback  Considerations 


Timeliness.  Deployed  operational  Army  as  well  as  Joint  and  Combined  headquarters  (HQs) 
generally  operate  7  days  a  week/ 12- 14  hours  a  day.  An  analytical  request  to/from  these 
headquarters  often  has  a  quick-turn  suspense  that  requires  an  answer  within  a  few  hours  or  days, 
generating  force  organizations  performing  reachback  support  must  adapt  to  these  "quick-turn" 
requirements.  Deployed  ORSA  analysts  must  adapt  to  the  environment  within  which  generating 
force  organizations  operate.  This  environment  requires  a  detailed,  time-intensive,  review  process 
to  ensure  a  high  quality  product. 
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Confidentiality  of  Derived  Analysis.  Generating  Foree  organizations  are  eustomer-focused. 
They  must  obtain  Operational  Army  Command  release  authority  before  disseminating  derived 
analytical  products.  This  release  protocol  ensures  customer  confidentiality  in  what  often 
concerns  contentious  issues. 

Organizational  Capabilities.  One  should  become  familiar  with  the  capabilities  of  Generating 
Force  organizations.  This  familiarity  enables  one  to  incorporate  current  organizational  efforts 
and  understand  where  a  problem  solution  may  reside. 

Contextual  Reference.  One  must  clearly  articulate  one's  analytical  requirements  when 
requesting  reachback  support.  A  well-articulated  request  ensures  that  reachback  personnel 
address  the  problem  within  the  context  of  other  competing  issues  such  as  time  constraints,  level 
of  model  sophistication,  and  the  essential  elements  of  analysis. 

8.4  Reachback  Support  Topics 


Generating  Force  analytical  support  to  the  current  fight  has  thus  far  included  campaign  plan 
assessments,  casualty  analyses,  theater  campaign  modeling  and  analyses,  force-on-force 
analyses,  stability  and  support  operations  analyses,  and  weapon  systems  analyses.  ORSA  topics 
span  doctrine,  organizations,  training,  materiel,  leadership  and  education,  personnel,  and 
facilities  (DOTMLPF).  Examples  of  reachback  support  topics  include: 

•  Campaign  plan  assessments.  Campaign  plan  assessments  include  the  development  of 
appropriate  MOEs  and  MOPs. 

•  Casualty  analyses.  Casualty  analyses  include  efficacy  of  personal  equipment  and 
implications  for  operating  force  TTPs. 

•  Theater  campaign  modeling  and  analysis.  Theater  campaign  modeling  focuses  on  the 
Joint  and  Combined  operational/strategic  environment.  These  models  incorporate 
weapons  effectiveness  data,  unit  formations,  and  current  war  plans  in  support  of 
campaign  analyses.  Additionally,  air  and  missile  defenses,  along  with  weapons  of  mass 
destruction  (chemical,  biological,  radiological,  and  nuclear),  are  analyzed  at  the  tactical, 
operational,  and  strategic  levels  of  war.  The  Total  Army  Analysis  (TAA)  process  uses 
analytical  output  to  determine  future  force  requirements  for  the  Army. 

•  Force-on-force  modeling  and  simulation.  Generating  Eorce  ORSA  analysts  develop 
and  maintain  a  class  of  warfighting  force-on-force  models  and  simulations  (M&S) 
ranging  from  individual  objects,  e.g..  Soldiers,  weapons,  terrain  features  to  theater-level 
models  that  often  aggregate  objects,  e.g.,  battalions  at  Corps  level.  AMSAA  leads  Army 
efforts  in  modeling  platform  performance  parameters.  TRAC  focuses  on  Corps  and 
below  force-on-force  combat  models  while  CAA  conducts  theater-level  campaign 
analysis.  Collectively,  these  models  are  widely  used  by  DOD,  industry,  and  allied 
nations. 

•  Stability  and  support  operations.  Generating  Eorce  ORSA  organizations  contribute  to 
stability  operations  with  a  variety  of  analytical  products,  e.g.,  campaign  plan  assessment 
methods  development,  convoy  protection  analysis,  sensor  placement  recommendations, 
basing  analysis,  medical  asset  allocation  recommendations,  new  materiel  fielding  and 
utilization  analysis.  Intelligence  Preparation  of  the  Battlespace  (IPB)  assistance,  attack 
pattern  analysis,  economic  forecasting,  and  force  structure/size  recommendations.  The 
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foregoing  ORSA  capabilities  assist  in  promoting  a  secure  environment  and  aid  in 
diplomatic  and  economic  programs  designed  to  eliminate  root  causes  of  instability. 

These  contributions  complement  and  reinforce  overall  Stability  and  Support  Operations 
(SASO). 

•  Weapon  Systems  analysis.  Weapons  Systems  analysis  enables  an  understanding  of  the 
system,  its  functions,  performance  and  effectiveness  measures,  and  operational  criteria. 
The  ORSA  community  can  provide  a  comprehensive  assessment  of  the  impact  of  a  given 
friendly  or  enemy  weapons  system  on  the  operational  environment.  The  Army  Test  and 
Evaluation  (ATEC)  Command  is  one  Generating  Eorce  organization  with  keen  weapon 
systems  insights. 

•  Programmatic  analysis.  The  cost  of  campaigns  and  major  operations  strongly  affects 
domestic  support  and  helps  determine  success  or  failure  of  a  given  mission.  ORSA 
analysts  aid  in  decision  making  with  analysis  of  logistics,  force  structure,  and 
mobilization/deployment  simulation  modeling.  These  projects  include  force  closure 
estimates,  resources  for  mobilization/deployment  data,  lift  asset  requirements,  pre¬ 
positioning  recommendations,  and  high-level  quick-turn  COA  analysis.  ORSA  analysts 
conduct  analysis  of  support  forces,  i.e.,  combat  support  and  combat  service  support 
requirements  and  casualty  analysis,  develop  wartime  Class  V  and  Class  Vll  requirements, 
and  compare  theater  logistics  requirements  with  capabilities. 

8.5  The  Reachback  Process 


General, 

Each  Generating  Eorce  organization  manages  its  respective  reachback  mechanisms 
independently.  CAA  conducts  an  informal  biweekly  interagency  meeting  with  deployed  ORSA 
analysts  in  an  effort  to  coordinate  reachback  efforts.  While  there  is  an  effort  to  consolidate  and 
formalize  reachback  contributions  from  Generating  Eorce  organizations,  this  chapter  focuses  on 
the  overall  reachback  process. 

Army  Materiel  System  Analysis  Agency  (AMSAA).  AMSAA  conducts  analyses  across  the 
Materiel  Eifecycle  to  inform  critical  decisions  for  current  and  future  Warfighter  needs  while 
valuing  the  unique  knowledge,  experiences,  and  backgrounds  of  its  people. 

Army  Test  and  Evaluation  Command  (ATEC).  ATEC  plans,  integrates,  and  conducts 
experiments,  developmental  testing,  independent  operational  testing,  and  independent 
evaluations  and  assessments  to  provide  essential  information  to  acquisition  decision  makers  and 
commanders. 

Center  for  Army  Analysis  (CAA).  CAA  conducts  analyses  across  the  spectrum  of  conflict  in  a 
Joint,  Interagency,  Intergovernmental,  and  Multinational  (JllM)  context  to  inform  critical  senior- 
level  decisions  for  current  and  future  national  security  issues.  CAA  supports  DA  and  its  major 
subordinate  commands  with  analyses  of  Army  forces  and  systems  within  the  context  of  joint  and 
combined  warfighting.  CAA  analyzes  strategic  concepts  and  military  options  and  estimates 
requirements  to  support  Army  input  to  the  Planning,  Programming,  Budgeting,  and  Execution 
System  (PPBES).  CAA  evaluates  (1)  the  Army's  ability  to  mobilize  and  deploy  forces,  (2)  Army 
force  capabilities,  (3)  force  alternatives,  and  (4)  theater  force-level  scenarios  and  resource 
analyses. 
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Marine  Corps  Combat  Development  Command  (MCCDC).  MCCDC  fully  integrates  Marine 
Corps  eoneepts  and  requirements  based  warfighting  capabilities;  including  doctrine, 
organization,  training,  materiel,  leadership  and  education,  personnel  and  facilities  in  order  to 
ensure  the  Marine  Corps  is  properly  organized,  trained  and  equipped  now  and  in  the  future. 

U,S.  Army  Training  and  Doctrine  Command  Analysis  Center  (TRAC).  TRAC  serves  as  the 
principal  analytical  organization  of  TRADOC  and  provides  centralized  leadership  and 
management  of  analysis  for  combat,  training,  and  doctrinal  developments,  while  unaligned  with 
TRADOC  proponents.  TRAC  conducts  studies  and  analyses  for  TRADOC  and  Headquarters, 
Department  of  the  Army  (HQ DA);  conducts  studies  of  the  integrated  battlefield  related  to 
doctrine,  organization,  training,  materiel,  personnel,  and  leadership;  designs  and  develops 
models  and  simulations  (M&S)  for  capabilities  development;  participates  in  technical  exchange 
programs  at  the  national  and  international  levels;  provides  analytical  support  to  Army 
Capabilities  Integration  Center  (ARCIC),  Centers  of  Excellence  (CoEs),  and  schools;  directs 
research  related  to  methods,  models,  tools,  and  analysis;  establishes,  maintains,  and  manages  the 
databases,  scenarios,  models,  and  wargaming  tools  required  to  support  analyses  and  studies;  and 
reviews  and  ensures,  as  directed,  the  quality  of  TRADOC  studies  before  their  approval. 

Non-ORSA  Generating  Force  organizations.  The  following  Generating  Eorce  organizations 
also  provide  relevant  information  to  many  ORSA  efforts: 

•  National  Ground  Intelligence  Center  (NGIC).  NGIC  provides  All  Source  and 
Geospatial  Intelligence  on  foreign  ground  force  capabilities  and  related  military 
technologies  and  integrates  with  Mission  Partners  to  ensure  the  U.S.  Army,  DoD,  Joint, 
and  National  level  decision  makers  maintain  decision  advantage  to  protect  the  U.S.  and 
interests  abroad. 

•  Joint  Improvised-Threat  Defeat  Agency  (JIDA).  JIDA  enables  Department  of 
Defense  actions  to  counter  improvised  threats  with  tactical  responsiveness  and  through 
anticipatory,  rapid  acquisition  in  support  of  Combatant  Commands'  efforts  to  prepare  for, 
and  adapt  to,  battlefield  surprise  in  support  of  counter-terrorism,  counter-insurgency,  and 
other  related  mission  areas  including  counter-improvised  explosive  device. 


Communication  Methods.  Deployed  ORSA  analysts  and  Generating  Eorce  organizations 
communicate  using  the  Defense  Switched  Network  (DSN),  commercial  telephone,  Non-secure 
Internet  Protocol  Router  Network  (NIPRNET),  Secret  Internet  Protocol  Router  Network 
(SIPRNET),  Secure  Video  Telephone  Conference  (SVTC),  Information  Work  Space  (IWS),  and 
other  means  beyond  the  scope  of  this  chapter.  Periodic  communications  ensure  established 
relationships  between  deployed  ORSA  analysts  and  their  respective  organizations.  Unclassified 
contact  information  for  the  most  commonly  requested  reachback  Generating  Eorce  organizations 
follows: 

•  Army  Logistics  University  (ALU)  College  of  Professional  and  Continuing 
Education.  Available  from  http://www.alu.army.mil/;  Commercial:  (804)765- 
4710/4991/4553  or  DSN  (539)  4710. 

•  Army  Materiel  Systems  Analysis  Activity  (AMSAA).  Available  from 
http://www.amsaa.army.mil/home.html;  Commercial  (410)  278-6614  or  DSN  (298)  6614. 
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•  Army  Test  and  Evaluation  Command  (ATEC).  Available  from 
http://www.army.mil/ATEC;  Commercial  (443)  861-9999  or  DSN  (848)  9999. 

•  Center  for  Army  Analysis  (CAA).  Available  from  http://www.caa.army.mil; 
Commercial  (703)  806-5580/5680  or  DSN  (656)  5580. 

•  Marine  Corps  Combat  Development  Command  (MCCDC).  Available  from 
http://www.mccdc.marines.mil;  Operations  Analysis  Directorate;  Commercial  (703)  784- 
6293  or  DSN  (278)  6293. 

•  National  Ground  Intelligence  Center  (NGIC).  Available  from 
https://www.inscom.army.mil/msc/NGlC.aspx;  Commercial  (434)  980-7000. 

•  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center  (TRAC).  Available 
from  http://www.trac.army.mil;  Commercial  (913)  684-9150  or  DSN  (552)  9150. 

8.6  Knowledge  Management 


Concept.  The  U.S.  Army  ORSA  community  provides  a  centralized  reachback  team,  trained  and 
equipped  to  coordinate  effective  and  timely  analytical  contributions  to  U.S.  Army  ORSA 
analysts  who  are  serving  within  deployed  elements  of  an  operational  headquarters,  e.g., 
Division/Corps/ASCC/Echelons  above  Corps  [EAC]/Joint/Combined).  CAA  OA  Division — 
with  participating  Generating  Eorce  organization  assistance — maintains  a  SharePoint  site 
(focused  on  current  operations)  on  Non-secure  Internet  Protocol  Router  Network  (NIPRNET)  to 
facilitate  deployed  ORSA  operations  and  analytical  reachback  support. 

Vision.  The  CAA  OA  Division  Current  Operations  Team  SharePoint  site  identifies,  creates, 
represents,  and  distributes  knowledge  for  reuse,  thereby  enabling  awareness  and  learning — ^with 
specific  emphasis  on  current  operations  support — across  the  ORSA  community.  Key  objectives 
of  this  site  include  shared  intelligence,  improved  performance,  competitive  advantage  over 
adversaries,  and  higher  levels  of  innovation. 

Structure.  Eigure  34  shows  the  general  structure  of  the  CAA  Operations  Analysis  (OA) 
Division  NIPRNET  SharePoint  site  available  from 

https://army/deps.mil/army/cmds/CAA/OA/SitePages/Home.aspx.  CAA  designed  this 
SharePoint  site  to  facilitate  coordination  between  analytical  agencies  and  deployed  ORSA 
analysts  regarding  reachback  and  Irregular  Warfare  (IW)  projects  and  data  issues.  Eeatured 
information  includes  organization  points  of  contact  (POCs),  a  request  form  for  reachback 
support,  and  contact  information  for  all  known  deployed  ORSA  analysts. 
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Operations  Analysis  (OA)  •  Home 

CAA  Command  Group  Strategic  Inibatives  Group* 

Deployment  &  Enablers  (DE)  *  Force  Strategy  (FS)  * 

Chief  Data  Officer  Chief  Information  Officer  Chief  Learning  Officer  Strategic  Partners  Army  Modeling  Su  Simulation  Office  (JWSO)  *  Campaign  Analysis  (CA)  * 

G  Staff-  Operations  Analysis  (OA)  *  Resource  Analysis  (RA)  *  Others  CAA  Activities/Events  Projects  Army  Operations  Research  Symposium  (AORS) 

1  Search  this  site... 

_Pi  Q 

View  /  Update 

OA  Project  Tracke 
OA  LEAVE.TDY 
CAA  Reachback 
OA  EOD  Checker 


L^raries 
Site  Pages 
Shared  Documents 
Document  Templates 
Deployment  Slate 

Usts 

Calendar 


Submit  o  reachback  request 
Submit  Leave  / Pass/ TDY 


Division  Chief:  COL  John  C.  Jackson 
View  Bio 

Deputy  Division  Chief:  vacant 
Site  Owner: 

Mr.  Neil  L.  de  Lara,  703-806-5063 


Discussions 
Team  Discussion 

OA  Leadership 
Under  Construction  O  Full  Name 

COL  John  Jackson 

^^Recyde  Bin  Mr.  Ron  Kollhoff 

All  Site  Content  Mr.  vem  Bahm 

Mr,  Mike  Stollenwerk 


Update  OA  Project  Status 


Division  Chief 

CUOPS  Analysis  Branch  Chief 

Functional  Aniysis  &  Assessment  Branch  Chief 

CUOPS  Support  Branch  Chief 


(703)  SOG-SSSO 
(703)  006-5079 
(703)  B0G-GG42 
(703)  SOG-5519 


E-mail  Address 

John.C.Jackson44.miiemBit.mil 
Ronald.K.Kollhoff2.cfvemail.mil 
Vernon  .J.Bahm.civOm  ail, mil 
Michael. F-Stollenwerlcav  email  .mil 


Mission;  OA  Division  deploys  trained  Operations  Research  Systems  Analysts 
(ORSA)  and  provides  responsive,  relevant,  and  credible  current  operations  reach- 
back.  irregular  war^re.  and  assessment  related  analyses  to  support  the  Army 
•  Q  staff.  Army  Seruce  Component  Commands,  and  Combatant  Commands  to  inform 
-  Joint,  combined,  and  interagency  senior-level  guidance,  planning,  and  decisions. 

Core  Capabilities: 

•  Select,  train,  and  deploy  analysts. 

•  Reach-back  analysis. 

•  Irregular  Warfare  (IW)  analysis. 

»  Assessment  expertise. 

•  Data  management  and  analysis. 

•  OR  institutional  development. 


Figure  34,  CAA  Operations  Analysis  Division  SharePoint  Site 

Reachback  Request  Form,  Figure  35  shows  the  structure  of  the  Request  for  Reachback  Form 
that  can  be  accessed  through  SharePoint  site  at  http://www.caa.army.mil  with  U.S.  CAC  login. 


Opersbons  Analysis  (OA)  •  Reach-badc  status.  New  Item 

CM  Command  Grco; 

1  Stralag<  Iniliativas  aro<4>*  CTeaf  Data  Offiesr  Ouaf  Intormalien  Ofhear  CNaf  Laame^  OMicer  Stratagac  Paitnari  Army  Madalir>e  A  Simwlat-en  Qfhot  (AMSOj  *  Campaign  Aoatysd  (CAj  *  - 

^  0 

Oaplovwant  B  EnaM's  ( 

DC'*  Forca  Strategy  (fS)  •  G  Staff  •  OparaMne  Analyaw  (OA)  • 

Raaei^ca  AnalyM  (RA)  •  Others  CM  AArnMa/Cvants  Prejaeti 

VMW 

OA  Prejaa  Tracker 

Raach-baek 

leave- Paas-Tpv 

Reach-back  Request  Form 

Librann 

Srte  Pages 

Shared  Documents 

Document  Templates 

Deptoymem  siata 

Calendar 

Tasks 

Reach  baiR  TMe 

Reach  hath  Orwcipitan 

Oiscwaswna 

Reauectec't  Marne 

Under  Construction 

Project  Status 

Requeiter'c  Emai 

Reauertac't  Phone 
(MN/COMJ. 

Reewecter’t  Cammand' 

Racyde  Bai 

4>  Ad  Sea  Ceetaie 

tleaiWcattan-  umciASS  {£_  Reachback  Anslyhcal 

Type 

OedcedtactetOale;  .3  *  cntkh.-.toatta<h.ftk 

Sutawt 

No4t  the  form  ¥110  tioie  after  eM»r>9SiJ9*MT 


Figure  35,  Reachback  Request  Form 


8.7  Formulating  Reachback  Problems 


In  general,  Generating  Force  organizations  should  adapt  to  the  requirements  of  deployed  ORSA 
analysts,  especially  in  terms  of  the  timeliness  of  derived  products.  To  aid  Generating  Force 
organization  endeavors,  inquiries  for  reachback  support  should  include  the  following  information 
in  as  much  detail  as  possible: 
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•  Problem  Statement.  One  should  provide  a  elear  definition  of  the  problem  along  with 
essential  elements  of  analysis.  In  particular,  what  question  should  reachback  answer? 
One  should  provide  a  description  of  all  assumptions  related  to  the  problem.  If  possible, 
one  should  describe  potential  modeling  methods. 

•  Problem  Background.  One  should  explain  factors/issues  bearing  on  the  problem.  This 
guides  one's  literature  review  and  provides  insight  to  reachback  personnel.  One  should 
describe  the  ultimate  customers  and  how  this  reachback  effort  supports  their  decision 
making  process  and  how  it  relates  to  other  theater  efforts,  i.e.,  issues  surrounding  the 
analysis. 

•  Data.  One  should  provide  data  or  recommendations  on  how  to  generate/obtain  the  data 
necessary  for  the  analysis. 

•  Timeline  and  suspense.  One  should  specify  desired  completion  dates  and  the  impact  of 
delayed  analysis  (details  on  intermediate  milestones  aid  in  project  management  and 
ensure  customer  satisfaction). 

•  Communications.  One  should  establish  recurrent  communication  with  the  supporting 
agency  via  the  CAA  Current  Operations  Team  SharePoint  site  on  NIPRNET  and  other 
communication  tools,  e.g.,  SIPRNET. 

•  Classification.  The  deployed  analyst  must  be  the  expert  as  to  the  classification  of 
(typically  coalition  derived)  data  and  products,  and  be  able  to  explain  to  others  how  the 
classification  relates  to  the  classifications  of  a  home  nation  and  other  relevant 
international  organizations,  e.g.,  the  North  Atlantic  Treaty  Organization  (NATO). 

•  Releasability.  One  should  specify  whether  the  customer  has  approved  the  release  of 
derived  analytical  products  developed  for  them.  Analytical  organizations  must  maintain 
customer  confidentiality;  however,  release  of  analytical  products  of  current  issues  and 
trends — where  possible — furthers  understanding  within  the  ORSA  community. 

8.8  Coordinating  Intra-/Inter-Theater  Reachback  Efforts 


Documenting  Analyses.  Proper  documentation  of  ongoing  and  completed  analyses  provides  an 
initial  start  point  for  new  analysis  and  reduces  duplication  of  efforts.  The  CAA  Current 
Operations  Team  asks  deployed  ORSA  analysts  to  provide  a  concise  briefing  (normally  one  or 
two  slides)  on  a  biweekly  basis  to  the  CAA  Current  Operations  meeting.  This  meeting  focuses 
on  current  ORSA  projects,  ongoing  reachback  projects,  and  other  miscellaneous  activities  that 
support  the  warfighter. 

•  Sharing  Data.  A  mechanism  for  sharing  very  large  unclassified  data  files  is  the  U.S. 
Army  Aviation  and  Missile  Research  Development  and  Engineering  Center  (AMRDEC) 
site  that  can  be  accessed  at  https://safe.amrdec.army.mil/safe.  AMRDEC  allows 
Common  Access  Card  (CAC)  holders  to  send  files  over  AMRDEC  SAFE  (Safe  Access 
File  Exchange),  and  also  allows  non-CAC  holders  to  send  files  to  CAC  holders.  On  the 
SIPRNET,  large  data  files  can  be  shared  through  the  U.S.  Air  Forces  Central  Command 
(USAFCENT)  Large  File  Distribution  System  (LEDS)  at 

https://lfds.shaw.afcent.af.smil.mil/default.aspx.  Headquarters  Resolute  Support  (RS) 
command  also  uses  a  system  called  Centaur  to  allow  specially  designated  and  trained 
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U.S.  staff  officers  and  analysts  to  quickly  move  UP  TO  RESOLUTE  SUPPORT^i  (RS) 
SECRET  files  from  the  RS  Mission  SECRET  network  to  SIPRNET  network,  and  from 
the  SIPRNET  network  to  the  RS  Mission  SECRET  network.  Additionally, 
USCENTCOM  provides  a  two-person  review  air  gapping  service  to  transfer  files  from 
SIPRNET  and  ISAE/SECRET  or  CENTRIXS  to  SIPRNET,  NIPRNET,  or 
ISAE/SECRET  or  CENTRIXS  -  the  files  must  be  marked  with  proper  classifications  on 
every  page  or  sheet.  The  files  must  be  emailable  and  it  takes  a  couple  days  for 
USCENTCOM  to  send  the  files  back  to  you  -  or  -  embarrassingly,  tell  you  that  your  files 
are  not  marked  properly  or  simply  of  the  wrong  classification  to  move  to  destination 
network  you  requested.  Here  are  the  links  you  need: 

o  Moving  files  from  SIPRNET  to  NIPRNET,  or  RS  SECRET  or  CENTRIXS: 
http://www.centrixs.centcom.smil.mil/CETS.cfm 

o  Moving  files  from  RS  SECRET  or  CENTRIXS:  http://22. 1 1 .80. 14/cfts/cfm;  use 
http://22.1 1.80.14/CETS.cfm  if  the  former  link  does  not  work. 

Generating  Eorce  organizations  alleviates  many  of  the  issues  with  data  storage  and  analysis. 

Data  management  and  analysis  continue  to  be  a  source  of  reachback  contributions. 


Recurrent  Coordination  Meetings.  Recurrent  meetings  provide  coordination  and  collaboration 
between  Generating  Eorce  organizations  and  deployed  ORSA  analysts.  Continued  support  of 
formal  and  informal  coordination  meetings  enables  effective  collaboration. 

Knowledge  Management,  The  CAA  OA  Division  Current  Operations  SharePoint  site  provides 
a  mechanism  to  manage  creation,  identification,  accumulation,  and  application  of  knowledge  for 
the  Operations  Research  (OR)  community.  This  effort  brings  together  inherent  intellectual 
capital. 

8.9  Pre-Deployment  Training  Opportunities  for  Deploying  Analysts 


Deploying  analysts  as  well  as  organizations  deploying  ORSA  analysts  to  conflict  areas  should 
work  in  advance  to  identify  and  resource  applicable  tailored  training  opportunities  for  each 
deployer.  The  following  known  training  opportunities  provide  ORSA  analysts  an  initial  menu  of 
courses  to  consider: 

•  The  CAA  hosted  no-fee  Deploying  Analyst  Course  (DAC),  offered  twice  yearly 
(currently  March  and  September)  at  Port  Belvoir,  VA;  POCs  at 
http://www.caa.army.mil/Commanders_Trifold_Jan20 1 6.pdf. 

•  The  Individual  Terrorism  Awareness  Course  (INTAC),  provided  by  the  Army  Corps  of 
Engineers  in  Winchester,  VA,  POC  Bill  Bicking  at  Bill.Bicking@usace.army.mil. 

•  Combined  Information  Data  Network  Exchange  (CIDNE)  &  Web-Enabled  Temporal 
Analysis  System  (WebTas);  POC  Jason  Thornton,  ISS  Corp.,  Alexandria,  VA  at 

J  ason.Thomton@issinc  .com. 


21  HQ  Resolute  Support  (RS)  is  the  successor  coalition  organization  that  replaced  HQ 
International  Security  Eorce  (ISAP)  on  1  JAN  2015. 
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•  Training  offered  by  the  deployers  future  deployed  eommand  hosted  in  the  Continental 
United  States. 

•  Geospatial  analysis  training  hosted  by  the  National  Geospatial  Agency  (NGA);  see 
course  opportunity  information  at  section  7.4,  above. 

•  Contracting  Officer's  Representative  (COR)  training;  see  Defense  Acquisition  University 
(DAU)  website  at  http;//icatalog.dau.mil/onlinecatalog/courses.aspx?crs_id=1731. 

•  NATO  related  training  opportunities  at  the  NATO  Joint  Force  Training  Centre  (JFTC) 
Bydgoszcz. 

•  The  Operations  Assessment  Course  at  the  NATO  School  in  Oberammergau,  Germany. 
See  https://www.natoschool.nato.int/Academics/Resident-Courses/Course-Catelog; 

Enrollment  POC  is  Boyd  Forbes  (Forbes. Boyd@natoschool.nato.int). 

•  For  assistance  identifying  viable  pre-deployment  for  deploying  analysts,  contact  the  CAA 
Operations  Analysis  (OA)  reachback  team  POCs  listed  at 
http://www.caa.armv.mil/Commanders  Trifold  Jan2016.pdf. 
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APPENDIX  A  LIST  OF  ACRONYMS 


AAG 

Afghan  Assessment  Group 

ACOM 

Army  Command 

AMSAA 

Army  Materiel  Systems  Analysis  Activity 

ANA 

Afghan  National  Army 

ANDP 

Afghan  National  Development  Program 

ANSF 

Afghan  National  Security  Forces 

AOR 

area  of  responsibility 

ArcGIS 

Arc  Geographic  Information  System 

ASCC 

Army  Service  Component  Commands 

ASCII 

American  Standard  Code  for  Information  Interchange 

ATEC 

Army  Testing  and  Evaluation  Command 

CAA 

Center  for  Army  Analysis 

BPS 

Bangsamoro  Perception  Survey 

CENTRIX 

Combined  Enterprise  Regional  Information  Exchange  System 

CE 

coalition  forces 

CIDNE 

Combined  Information  Data  Network  Exchange 

CIO 

Chief  Information  Office 

CJSOTE-A 

Combined  Joint  Special  Operations  Task  Eorce  -  Afghanistan 

CJTE-HOA 

Combined  Joint  Task  Eorce  -  Horn  of  Africa 

CLA 

Constraints,  Limitations,  and  Assumptions 

COA 

course  of  action 

CoEs 

Centers  of  Excellence 

COR 

Contracting  Officer's  Representative 

DA 

Department  of  the  Army 

DA  PAM 

Department  of  the  Army  Pamphlet 

DAC 

Deploying  Analyst  Course 

DAHB 

Deployed  Analyst  Handbook 
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DAHR 

Deployed  Analyst  History  Report 

DAST 

Deployed  Analyst  Support  Team 

DAU 

Defense  Acquisition  University 

DOD 

Department  of  Defense 

DOTMLPF 

Doctrine,  Organizations,  Training,  Materiel,  Leadership,  Personnel,  and  Facilities 

DSN 

Defense  Switched  Network 

EAC 

Echelons  Above  Corps 

FA 

Functional  Area 

GIS 

Geospatial  Information  System 

HN 

Host  Nation 

HOA 

Horn  of  Africa 

HQ 

Headquarters 

HQDA 

Headquarters,  Department  of  Army 

ID 

Identification 

lED 

Improvised  Explosive  Device 

INTAC 

Individual  Terrorism  Awareness  Course 

INFORMS 

Institute  for  Operations  Research  and  the  Management  Sciences 

IPB 

Intelligence  Preparation  of  the  Battlespace 

IR 

Information  Requirement 

ISAF 

International  Security  Assistance  Force 

ISF 

Iraqi  Security  Forces 

ISR 

Intelligence,  Surveillance,  and  Reconnaissance 

IW 

Irregular  Warfare 

IWS 

Information  Work  Space 

JP 

Joint  Publication 

JSOTF-P 

Joint  Special  Operations  Task  Force  -  Philippines 

M&S 

Models  and  Simulations 

MCCDC 

Marine  Corps  Combat  Development  Command 

MDMP 

Military  Decision  Making  Process 
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MEDEVAC 

Medical  Evacuation 

MNE-I 

Multi-National  Forces  -  Iraq 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

NATO 

North  Atlantic  Treaty  Organization 

NGIC 

National  Ground  Intelligence  Center 

NGA 

National  Geospatial  Agency 

NGO 

Non-Governmental  Organization 

NIPRNET 

Non-secure  Internet  Protocol  Router  Network 

NPS 

Naval  Post  Graduate  School 

OA 

Operations  Analysis 

OCO 

Overseas  Contingency  Operation 

ODBC 

Open  Database  Connectivity 

OE 

Operational  Environment 

OEE 

Operation  Enduring  Freedom 

OES 

Operation  Freedom  Sentinel 

OIE 

Operation  Iraqi  Freedom 

OIR 

Operation  Inherent  Resolve 

OLAP 

Online  Analytical  Processing 

OLE  DB 

Object  Linking  and  Embedded  Database 

OMB 

Office  of  Management  and  Budget 

OND 

Operation  New  Dawn 

OR 

Operations  Research 

ORSA 

Operations  Research/Systems  Analysis 

PMESII 

Political,  Military,  Economic,  Social,  Information,  and  Infrastructure 

POC 

Point  of  Contact 

PPBES 

Planning,  Programming,  Budgeting,  and  Execution  System 

P-Value 

Probability  Value 

RS 

Resolute  Support 
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SAS 

Statistical  Analysis  System 

SASO 

Stability  and  Support  Operations 

SIPRNET 

Secret  Internet  Protocol  Router  Network 

SPPPS 

Southern  Philippines  Public  Perception  Survey 

SPSS 

Statistical  Package  for  the  Social  Sciences 

TAA 

Total  Army  Analysis 

TRAC 

TRADOC  Analysis  Center 

TRADOC 

United  States  Army  Training  and  Doctrine  Command 

TTPs 

Tactics,  Techniques,  and  Procedures 

U.S. 

United  States 

USAID 

United  States  Agency  for  International  Development 

USMA 

United  States  Military  Academy 

VBA 

Visual  Basic  for  Applications 

VTC 

Video  Telephone  Conference 

WebTAS 

Web-Enabled  Temporal  Analysis  System 
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